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CLAIMS 



[Claim(s)] 

[Claim l] It is motion detection equipment which detects a motion of the image pick-up 
direction of said image pick-up equipment from the image picturized by image pick-up 
equipment. An are recording image storage means to memorize the are recording image 
in the condition of having piled up by performing alignment of the background of each 
screen of the image which said image pick-up equipment outputs, A detection entry 
means to set up the detection range used for detecting said motion about said attention 
screen containing the image of the screen in front of the attention screen in said are 
recording image, A center-of- gravity calculation means to compute the center of gravity 
of said detection range, and the center of gravity of said attention screen, A motion 
vector calculation means to ask for the motion vector which expresses a motion of the 
image pick-up direction of said image pick-up equipment when picturizing the image of 
said attention screen based on the center of gravity of said detection range, and the 
center of gravity of said attention screen, Motion detection equipment characterized by 
including the write-in means which writes the image of said attention screen in said are 
recording image storage means based on said motion vector. 

[Claim 2] Said center-of-gravity calculation means is motion detection equipment 
according to claim 1 characterized by classifying the pixel in said detection range or an 
attention screen into either of some level based on the pixel value, computing the center 
of gravity of the pixel belonging to each level, and searching for the center of gravity of 
the center of gravity of each level as a center of gravity of said detection range or an 
attention screen, respectively. 

[Claim 3] Said center-of-gravity calculation means is motion detection equipment 
according to claim 1 characterized by classifying each pixel in said detection range or an 
attention screen into either of some classes based on the pixel around it, computing the 
center of gravity of the pixel belonging to each class, and searching for the center of 
gravity of the center of gravity of each class as a center of gravity of said detection range 
or an attention screen, respectively. 

[Claim 4] Said center-of-gravity calculation means is motion detection equipment 
according to claim 3 characterized by computing the center of gravity of the pixel 
belonging to a specific class, and searching for the center of gravity of the center of 
gravity of said specific class as a center of gravity of said detection range or an attention 
screen, respectively. 
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[Claim 5] Said center-of-gravity calculation means is motion detection equipment 
according to claim 4 characterized by searching for the center of gravity of said 
detection range or an attention screen for the class to which the pixel of the part which 
is an edge belongs as said specific class, respectively. 

[Claim 6] It is motion detection equipment according to claim 1 characterized by to ask 
for said motion vector based on the center of gravity of said detection range, and the 
center of gravity of said attention screen only when judged with said motion vector 
calculation means having a motion in said motion existence judging means, including 
further a motion existence judging means judge the existence of a motion of the image 
pick-up direction of said image pick-up equipment when picturizing the image of said 
attention screen. 

[Claim 7] Said motion existence judging means is motion detection equipment according 
to claim 6 characterized by classifying each pixel in said attention screen into either of 
some classes based on the pixel around it, measuring the pixel belonging to a specific 
class, and the pixel to which the screen in front of said attention screen corresponds, 
and judging the existence of said motion based on the comparison result. 
[Claim 8] Said motion existence judging means is motion detection equipment according 
to claim 7 characterized by measuring the pixel which belongs to the specific class 
considering the class to which the pixel of the part which is an edge belongs as said 
specific class, and the pixel to which the screen in front of said attention screen 
corresponds. 

[Claim 9] It is the motion detection approach of detecting a motion of the image pick-up 
direction of said image pick-up equipment from the image picturized by image pick-up 
equipment. The inside of said are recording image memorized by are recording image 
storage means to memorize the are recording image in the condition of having piled up 
by performing alignment of the background of each screen of the image which said 
image pick-up equipment outputs, The detection entry step which sets up the detection 
range used for detecting said motion about said attention screen containing the image of 
the screen in front of an attention screen, The center-of-gravity calculation step which 
computes the center of gravity of said detection range, and the center of gravity of said 
attention screen, The motion vector calculation step which asks for the motion vector 
which expresses a motion of the image pick-up direction of said image pick-up 
equipment when picturizing the image of said attention screen based on the center of 
gravity of said detection range, and the center of gravity of said attention screen, The 
motion detection approach characterized by including the write-in step which writes the 
image of said attention screen in said are recording image storage means based on said 
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motion vector. 

[Claim 10] The program for performing motion detection processing in which a motion 
of the image pick-up direction of said image pick-up equipment is detected, from the 
image picturized by image pick-up equipment The inside of said are recording image 
memorized by are recording image storage means to have been the medium which a 
computer is made to perform and to memorize the are recording image in the condition 
of having piled up by performing alignment of the background of each screen of the 
image which said image pick-up equipment outputs, The detection entry step which sets 
up the detection range used for detecting said motion about said attention screen 
containing the image of the screen in front of an attention screen, The center-of-gravity 
calculation step which computes the center of gravity of said detection range, and the 
center of gravity of said attention screen, The motion vector calculation step which asks 
for the motion vector which expresses a motion of the image pick-up direction of said 
image pick-up equipment when picturizing the image of said attention screen based on 
the center of gravity of said detection range, and the center of gravity of said attention 
screen, The medium which makes said computer execute the program characterized by 
including the write-in step which writes the image of said attention screen in said are 
recording image storage means based on said motion vector. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention] Especially this invention relates to a medium at the motion 
detection equipment and the motion detection approach of enabling it to detect 
simply a motion (suitably henceforth a camera motion) of the video camera, and a list 
from the image picturized by motion detection equipment and the motion detection 
approach, and the list with image pick-up equipments, such as a video camera, about 
the medium. 
[0002] 

[Description of the Prior Art] As an approach of detecting a camera motion, the same 
with detecting the so-called motion vector of a macro block, it considers that the 
whole frame is one block, and how to perform block matching can be considered, for 
example. 
[0003] 
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[Problem(s) to be Solved by the Invention] however — in performing block matching 
for the whole frame, while shifting the spatial location of the frame which is going to 
detect the camera motion — an absolute value with the pixel of the image in front of 
one of them — since it is necessary to ask for the total about one frame of difference 
etc., the amount of operations is size and troublesome. 

[0004] This invention is made in view of such a situation, and enables it to ask for a 

camera motion simply. 

[0005] 

[Means for Solving the Problem] An are recording image storage means to memorize 
the are recording image in the condition that the motion detection equipment of this 
invention piled up by performing alignment of the background of each screen of the 
image which image pick-up equipment outputs, A detection entry means to set up the 
detection range used for detecting the motion about the attention screen containing 
the image of the screen in front of the attention screen in an are recording image, A 
center-of-gravity calculation means to compute the center of gravity of the detection 
range, and the center of gravity of an attention screen, and the center of gravity of 
the detection range, Based on the center of gravity of an attention screen, it is 
characterized by including a motion vector calculation means to ask for the motion 
vector showing a motion of the image pick-up direction of the image pick-up 
equipment when picturizing the image of an attention screen, and the write-in means 
which writes the image of an attention screen in an are recording image storage 
means based on a motion vector. 

[0006] A center-of-gravity calculation means can be made to be able to classify the 
pixel in the detection range or an attention screen into either of some level based on 
the pixel value, and can be made to be able to compute the center of gravity of the 
pixel belonging to each level, and the center of gravity of the center of gravity of each 
level can be made to be searched for as a center of gravity of the detection range or 
an attention screen, respectively. Moreover, a center-of-gravity calculation means 
can be made to be able to classify each pixel in the detection range or an attention 
screen into either of some classes based on the pixel around it, and can be made to be 
able to compute the center of gravity of the pixel belonging to each class, and the 
center of gravity of the center of gravity of each class can be made to be searched for 
as a center of gravity of the detection range or an attention screen, respectively. 
Furthermore, a center-of-gravity calculation means can be made to be able to 
compute the center of gravity of the pixel belonging to a specific class, and the center 
of gravity of the center of gravity of a specific class can be made to be searched for 
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as a center of gravity of the detection range or an attention screen, respectively. 
Moreover, the center of gravity of the detection range or an attention screen can be 
made for the class to which the pixel of the part which is an edge belongs to a 
center-of-gravity calculation means to be searched for as a specific class, 
respectively. 

[0007] A motion existence judging means judge the existence of a motion of the image 
pick-up direction of the image pick-up equipment when picturizing the image of an 
attention screen can be established further, and only when judged with there being a 
motion in a motion vector calculation means in a motion existence judging means in 
this case by the motion detection equipment of this invention, it can be made to be 
asked for a motion vector based on the center of gravity of the detection range, and 
the center of gravity of an attention screen. 

[0008] A motion existence judging means can be made to be able to classify each pixel 
in an attention screen into either of some classes based on the pixel around it, the 
pixel belonging to a specific class and the pixel to which the screen in front of an 
attention screen corresponds can be made to be able to measure with it, and the 
existence of a motion can be made to judge based on the comparison result. Moreover, 
the pixel which belongs to the specific class considering the class to which the pixel of 
the part which is an edge belongs as a specific class, and the pixel to which the screen 
in front of an attention screen corresponds can be made to measure with a motion 
existence judging means. 

[0009] The inside of the are recording image memorized by are recording image 
storage means to memorize the are recording image in the condition that the motion 
detection approach of this invention piled up by performing alignment of the 
background of each screen of the image which image pick-up equipment outputs, The 
detection entry step which sets up the detection range used for detecting the motion 
about the attention screen containing the image of the screen in front of an attention 
screen. The center- of-gravity calculation step which computes the center of gravity 
of the detection range, and the center of gravity of an attention screen, The motion 
vector calculation step which asks for the motion vector which expresses a motion of 
the image pick-up direction of the image pick-up equipment when picturizing the 
image of an attention screen based on the center of gravity of the detection range, 
and the center of gravity of an attention screen, It is characterized by including the 
write-in step which writes the image of an attention screen in an are recording image 
storage means based on a motion vector. 

[0010] The program which the medium of this invention makes a computer execute 
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The inside of the are recording image memorized by are recording image storage 
means to memorize the are recording image in the condition of having piled up by 
performing alignment of the background of each screen of the image which image 
pick-up equipment outputs, The detection entry step which sets up the detection 
range used for detecting the motion about the attention screen containing the image 
of the screen in front of an attention screen, The center-of-gravity calculation step 
which computes the center of gravity of the detection range, and the center of gravity 
of an attention screen, The motion vector calculation step which asks for the motion 
vector which expresses a motion of the image pick-up direction of the image pick-up 
equipment when picturizing the image of an attention screen based on the center of 
gravity of the detection range, and the center of gravity of an attention screen. It is 
characterized by including the write-in step which writes the image of an attention 
screen in an are recording image storage means based on a motion vector. 
[001 1] It sets to a medium at the motion detection equipment of this invention and the 
motion detection approach, and a list. The inside of the are recording image 
memorized by are recording image storage means to memorize the are recording 
image in the condition of having piled up by performing alignment of the background of 
each screen of the image which image pick-up equipment outputs, The detection 
range used for detecting the motion about the attention screen containing the image 
of the screen in front of an attention screen is set up, and the center of gravity of the 
detection range and the center of gravity of an attention screen are computed. And 
based on the center of gravity of the detection range, and the center of gravity of an 
attention screen, the motion vector showing a motion of the image pick-up direction 
of the image pick-up equipment when picturizing the image of an attention screen is 
called for, and the image of an attention screen is written in an are recording image 
storage means based on the motion vector. 
[0012] 

[Embodiment of the Invention] Drawing 1 shows the example of a configuration of the 
gestalt of 1 operation of the image transmission equipment which applied this 
invention. 

[0013] The digital image data as an image (here, it considers as a dynamic image) set 
as the object of coding photoed with the video camera etc. are supplied, the image is 
encoded by the encoder 1 and the coded data obtained as a result is outputted to it 
there. This coded data is transmitted through the transmission media 3, such as a 
satellite circuit, and a ground wave, a CATV (Cable Television) network, the Internet, 
ISDN (Integrated Service Digital Network), or is recorded on the record media 4, such 
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as an optical disk, and a magneto-optic disk, a magnetic disk, a magnetic tape, a phase 
change disk. 

[0014] The coded data transmitted through a transmission medium 3 or the coded 
data reproduced from a record medium 4 is supplied to a decoder 2, in a decoder 2, 
the coded data supplied there is decoded, and the decode image obtained as a result 
is supplied and displayed on the monitor which is not illustrated, for example. 
[0015] The above image transmission equipment is applicable to the equipment which 
transmits and receives an image in the distant location, for example, the equipment 
which performs record playback of an image. 

[0016] Next, drawing 2 shows the example of a configuration of the encoder 1 of 
drawing 1 . 

[0017] The are recording section 11 consists of semiconductor memory, a magnetic 
disk, etc., and stores temporarily the image data supplied to an encoder 1. Here, image 
data is inputted and memorized by the are recording section 1 1 , for example per frame 
of a series of images, such as from a certain scene change until [ of a degree ] a scene 
change. In addition, as shown in drawing 3 , the image data of the N frame which 
becomes by the 1st frame thru/or the Nth frame shall be memorized by the are 
recording section 1 1 as a series of images here, for example. 

[0018] If the image of the N frame as a series of images is memorized by the are 
recording section 11, the camera motion detecting element 12 will read the image of 
the N frame from the are recording section 1 1 , and will detect the camera motion 
vector showing the direction and magnitude of a camera motion in each frame. 
[0019] That is, the camera motion detecting element 12 sets the camera motion 
vector v1 of the 1st frame as 0 (= (0 0)) in the system of coordinates (suitably 
henceforth standard coordinates) which make the right or down [ a top to ] a x axis or 
the y-axis from the left, respectively while making a zero the point at the upper left of 
the 1st frame, as shown in drawing 4 (A). And about the 2nd frame thru/or the Nth 
frame, the camera motion detecting element 1 2 searches for the coordinate (x y) of 
the criteria seat table system in which the point at the upper left of the n-th frame is 
located as the camera motion vector vn of the n-th frame, when alignment of the 
background of each frame is performed. 

[0020] After the camera motion detecting element 1 2 sets the camera motion vector 
v1 of the 1st frame as 0, as shown in drawing 4 (B), in standard coordinates, it asks for 
the location of the 1 st frame and the 2nd frame whose backgrounds correspond, and, 
specifically, searches for the coordinate of the point at the upper left of [ in the 
location ] the 2nd frame as the camera motion vector v2. Furthermore, as shown in 
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drawing 4 (C), in standard coordinates, the camera motion detecting element 1 2 asks 
for the location of the 1 st frame which performed alignment of a background and the 
2nd frame, and the 3rd frame whose backgrounds correspond, and searches for the 
coordinate of the point at the upper left of [ in the location ] the 3rd frame as the 
camera motion vector v3. 

[0021] Hereafter, the camera motion detecting element 12 calculates the camera 
motion vector v4 of the 4th frame thru/or the Nth frame thru/or vN similarly. 
[0022] In addition, in order to simplify explanation, as a camera motion, only a motion 
of a perpendicular direction is considered and let revolutions be level and the thing 
which is not considered here. However, this invention can be applied even when a 
camera motion has a revolution. 

[0023] The camera motion vector v1 of the 1st frame thru/or the Nth frame as a 
series of images detected by the camera motion detecting element 1 2 as mentioned 
above thru/or vN are supplied to the camera motion vector storage section 1 3, and is 
memorized. 

[0024] When the camera motion vector v1 thru/or vN are memorized, in the camera 
motion vector storage section 13 the background extract section 14 While reading the 
camera motion vector v1 thru/or vN from the camera motion vector storage section 
13 Read the image data of the 1 st frame thru/or the Nth frame from the are recording 
section 11, and by performing alignment of the background of the 1st frame thru/or 
the Nth frame based on the camera motion vector v1 thru/or vN the 1st frame 
thru/ or that the Nth frame background of the whole (this background — for example, 
a series of images — a video camera — panning — or if chill TINGU is carried out and 
a photograph is taken, it will become an oblong or longwise image) (suitably henceforth 
a whole background) are extracted. The whole background extracted in the 
background extract section 14 is supplied to the background memory 15, and is 
memorized. 

[0025] In the background memory 15, if a whole background is memorized, the 
foreground coding section 1 6 detects the background of each frame memorized by the 
are recording section 1 1 among the whole background based on the camera motion 
vector of each frame memorized by the camera motion vector storage section 1 3, will 
be subtracting the background of each of that detected frame from the image of each 
frame, and will extract the foreground of each frame. Furthermore, the foreground 
coding section 16 encodes the foreground of each frame, and outputs the coding 
result to MUX(multiplexer) 17. 

[0026] From the foreground coding section 16, if the coding result of a foreground is 
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received, MUX17 will multiplex the camera motion vector memorized by the camera 
motion vector storage section 13 and the whole background memorized by the 
background memory 1 5 to the coding result of the foreground, and will output the 
multiplexing result to it as coded data. 

[0027] In an encoder 1, image data is encoded per a series of images as mentioned 
above. 

[0028] Next, drawing 5 shows the example of a configuration of the camera motion 
detecting element 1 2 of drawing 2 . 

[0029] A series of images memorized by the are recording section 1 1 ( drawing 2 ) are 
supplied to the center- of-gravity calculation section 21 per frame, and the 
center-of-gravity calculation section 21 asks it for a center of gravity which is later 
mentioned about each frame. Furthermore, the center-of-gravity calculation section 
21 sets up the range (suitably henceforth the motion detection range) used for 
detecting the camera motion vector of the attention frame currently observed to the 
are recording image mentioned later memorized in the are recording image memory 24, 
and also searches for the center of gravity of the motion detection range. The center 
of gravity of the attention frame called for in the center-of-gravity calculation section 
21 and the motion detection range is supplied to the vector detecting element 22. 
[0030] The vector detecting element 22 is supplied to the write-in control section 23 
while it detects the camera motion vector of an attention frame and supplies it to the 
camera motion vector storage section 1 3 ( drawing 2 ) based on the center of gravity 
of the attention frame supplied from the center-of-gravity calculation section 21, and 
the motion detection range. 

[0031] The write-in control section 23 controls the address with which the are 
recording image memory 24 memorizes the image data of an attention frame based on 
the camera motion vector from the vector detecting element 22. From the are 
recording section 1 1 ( drawing 2 ), the are recording image memory 24 reads the image 
data of an attention frame, and memorizes it to the address specified by the write-in 
control section 23. 

[0032] Next, with reference to drawing 6 , the camera motion detection processing 
performed in the camera motion detecting element 12 of drawing 5 in which a camera 
motion vector is detected is explained. 

[0033] The point at the upper left of [ in the condition of having performed alignment 
of each frame and having performed the alignment ] each frame is detected as a 
camera motion vector of each frame so that the center of gravity of each frame may 
be in agreement fundamentally paying attention to the center of gravity of an image 
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moving the camera motion detection processing which the camera motion detecting 
element 1 2 performs by camera motion. 

[0034] That is, if the n-th frame is now used as an attention frame, the image in the 
condition of having performed alignment of the background of each frame in the are 
recording image memory 24, and having laid the 1 st thru/or the image data of the 
n-1 st frame which is a frame to the frame in front of an attention frame on top of it in 
the sequence (are recording image) is memorized. 

[0035] In this case, the center— of-gravity calculation section 21 searches for the 
center of gravity cn of the n-th frame which is an attention frame, as shown in drawing 
6 (A). Furthermore, as shown in drawing 6 (B), the center-of-gravity calculation 
section 21 makes the range which includes the n-1st frame in front of one of the 
attention frames in the are recording image memorized in the are recording image 
memory 24 the motion detection range, and searches for the center of gravity c of the 
motion detection range. Here, as motion detection range, the range where only the 
predetermined number of pixels is large is set up in each direction of four directions of 
the n-1 st frame, for example. 

[0036] By the vector detecting element 22, if the center of gravity cn of an attention 
frame and the center of gravity c of the motion detection range are searched for, as 
shown in drawing 6 (C), the location of the point at the upper left of [ in the condition 
that the center of gravity cn of an attention frame was in agreement with the center 
of gravity c of the motion detection range ] an attention frame will be called for, and 
the coordinate of the location will be outputted as a camera motion vector vn of the 
n-th frame which is an attention frame. 

[0037] That is, considering the n-th frame as an attention frame, in asking for the 
camera motion vector vn, it has already found the camera motion vector of a before 
[ one of them ]. Then, as shown in drawing 6 (C), while expressing the location on the 
basis of the point at the upper left of [ of the center of gravity c of the motion 
detection range ] the n-1st frame with vector v'c Supposing it expresses with vector 
v'cn the location on the basis of the point at the upper left of [ of the center of gravity 
cn of the n-th frame which is an attention frame / the ] the n-th frame The 
coordinate in the standard coordinates of the location of the point at the upper left of 
[ in the condition that the center of gravity c of the motion detection range and the 
center of gravity cn of an attention frame were in agreement ] an attention frame 
serves as the motion vector vn of an attention frame. And it can ask for this camera 
motion vector vn by subtracting vector v'cn which adds vector v'c showing the 
location of the center of gravity c of the motion detection range to motion vector 
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vn-1 in front of one attention frame of the n-1 st frame, and expresses the location of 
the center of gravity cn of an attention frame to it further. That is, it can ask for the 
camera motion vector vn of an attention frame by calculating formula 
vn=vn-1 +v'c-v'cn. 

[0038] After the camera motion vector vn of an attention frame is called for as 
mentioned above, in the write-in control section 23, the write-in address for writing in 
the image data of the attention frame in the are recording image memory 24 is 
controlled based on the camera motion vector vn. Namely, thereby, in standard 
coordinates, the image data of an attention frame is written in the point shown by the 
camera motion vector vn in the form to overwrite, and the image obtained as a result 
of the writing is used for it by using the following frame [ n+1 st ] as an attention frame 
in the are recording image memory 24 as an are recording image at the time of 
detecting the camera motion vector vn+1 so that the point of the upper left may be 
located. 

[0039] Next, with reference to the flow chart of drawing 7 , the camera motion 
detection processing in the camera motion detecting element 12 of drawing 5 is 
explained further. 

[0040] While reading appearance of the 1 st frame of a series of images memorized by 
the are recording section 1 1 is carried out as an attention frame and it is first supplied 
to the center— of-gravity calculation section 21, the storage value of the are recording 
image memory 24 is cleared. 

[0041] And in the center— of-gravity calculation section 21, it is judged in step S1 
whether there is any 1 st attention frame. In step S1 , when judged with there being the 
1st attention frame, it progresses to step S2, and the vector detecting element 22 
sets up 0 as the camera motion vector v1, outputs it to the camera motion vector 
storage section 1 3 and the write-in control section 23, and progresses to step S6. 
[0042] Based on the camera motion vector from the vector detecting element 22, the 
write-in control section 23 controls the write-in address in the are recording image 
memory 24 by step S6, and, thereby, writes an attention frame in the are recording 
image memory 24 at it. That is, in now, there is the 1 st attention frame, and since the 
camera motion vector v1 is 0, the image data of the 1 st frame is written in in the are 
recording image memory 24 so that the point of the upper left may be located in the 
zero in standard coordinates. 

[0043] Then, it progresses to step S7, and when judged with it being judged whether 
there is any following frame which constitutes a series of images, and occurring by the 
are recording section 11, reading appearance of the following frame is newly carried 
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out to it as an attention frame, and it is supplied to it at the center-of-gravity 
calculation section 21. And the same processing is repeated by step S1 return and the 
following. 

[0044] When it is judged with there not being the 1st attention frame in step S1 on the 
other hand (i.e., when it is either the 2nd frame or thru/or the Nth frame), it 
progresses to step S3, and in the center-of-gravity calculation section 21, 
center-of-gravity calculation processing in which the center of gravity of an attention 
frame is searched for is performed, and it progresses to step S4. In step S4, in the 
center— of-gravity calculation section 21, the motion detection range to an attention 
frame is set up into the are recording image memorized in the are recording image 
memory 24, center-of-gravity calculation processing in which the center of gravity of 
the motion detection range is searched for is performed, and it progresses to step S5. 
[0045] At step S5, in the vector detecting element 22, as drawing 6 explained, the 
camera motion vector of an attention frame is called for, and it is outputted to the 
camera motion vector storage section 1 3 and the write-in control section 23 from the 
center of gravity of the attention frame called for in the center-of-gravity calculation 
section 21, and the center of gravity of the motion detection range. 
[0046] And as progressed and mentioned above to step S6, in the write-in control 
section 23, based on the camera motion vector from the vector detecting element 22, 
the write-in address in the are recording image memory 24 is controlled, and, thereby, 
an attention frame is written in the are recording image memory 24. That is, in the are 
recording image memory 24, in standard coordinates, the image data of an attention 
frame is written in the point shown by the camera motion vector of an attention frame 
so that the point of the upper left may be located (overwritten). 

[0047] Then, as progressed and mentioned above to step S7, when it is judged 
whether the following frame which constitutes a series of images is in the are 
recording section 1 1 and it is judged with there being nothing (i.e., when the 1st which 
constitutes a series of images thru/or the camera motion vector of each Nth frame 
are called for), camera motion detection processing is ended. 

[0048] Next, with reference to the flow chart of drawing 8 , the processing 
(center-of-gravity calculation processing of an attention frame) which the 
center-of-gravity calculation section 21 performs is explained in full detail in step S3 
of drawing 7 . 

[0049] First, in step S1 1 , to Variables X or Y, the number of pixels beside an attention 
frame (horizontal direction) or the vertical (perpendicular direction) number of pixels is 
set, respectively, and progresses to step S12. 
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[0050] -1 as initial value is set to the variable y which expresses the y-coordinate of 
each pixel of an attention frame with step S12, it progresses to step S22, and the 
increment of the variable y is carried out only for 1. 

[0051] Here, the coordinate of each pixel which constitutes an attention frame makes 
the pixel on the leftmost a zero, shall take a x axis or the y-axis from the left from a 
top to the right or down, respectively, and shall express it. 

[0052] Then, it progresses to step S14 and it is judged whether Variable y is under the 
number Y of pixels of the length of an attention frame. In step S14, when judged with 
Variable y being under Y, it progresses to step S1 5, and -1 as initial value is set to the 
variable x showing the x-coordinate of each pixel of an attention frame, and it 
progresses to step S16. 

[0053] At step S16, the increment only of 1 is carried out, Variable x progresses to 
step S1 7, and it is judged whether Variable x is under the number X of pixels beside an 
attention frame. In step S1 7, when judged with Variable x not being under X, the same 
processing is repeated by step S13 return and the following. 

[0054] Moreover, in step S17, when judged with Variable x being under X, it progresses 
to step S18, the pixel p (x y) in a coordinate (x y) is made into an attention pixel, and 
the attention pixel is classified into either of the level set up beforehand based on the 
pixel value. 

[0055] That is, with the gestalt of this operation, the range of the value which can be 
taken as a pixel value is beforehand divided into some range. And if the range of the 
value which can be taken as a now, for example, pixel, value shall be divided into K 

range and this K range shall be called level 1, 2 K in an order from the range where 

a pixel value is small, at step S18, an attention pixel will be classified according to 
whether a pixel value belongs to level 1 thru/or which range of the K. 
[0056] Furthermore, the level classification result of an attention pixel is registered 
into a level table at step S18. 

[0057] The center-of-gravity calculation section 21 Namely, among those, the 
frequency fk of the pixel which belongs to the memory (not shown) to harbor at the 

level k about each level k (k= 1, 2 K) as shown in drawing 9 , When the level table 

which matched integrated value sigmaxk of the x-coordinate of the pixel belonging to 
level k and integrated value sigmayk of a y-coordinate is memorized, for example, the 
pixel value of an attention pixel belongs to level k While only 1 increments the 
frequency fk about the level k in a level table, the x-coordinate or y-coordinate of an 
attention pixel is added to integrated value sigmaxk of an x-coordinate, or integrated 
value sigmayk of a y-coordinate, respectively. 
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[0058] In addition, a level table is cleared by 0 whenever processing according to the 
flow chart of drawing 8 is started. 

[0059] And the same processing is repeated by step S16 return and the following. 
[0060] It processes by making into an attention pixel each pixel which constitutes an 
attention frame on the other hand when judged with Variable y not being under Y in 
step S14, and when all the pixels that constitute an attention frame are registered into 
a level table, it progresses to step S19 and the center of gravity of the pixel belonging 
to each level of a level table is searched for. That is, the coordinate (sigmaxk/fk, 
sigmayk/fk) as which the division of integrated value sigmaxk of the x-coordinate in 
each level k of a level table or each integrated value sigmayk of a y-coordinate is done 
in the frequency fk, and it is expressed in step S19 with the division value is searched 
for as a center of gravity of the pixel belonging to each level k. 

[0061] and the step S20 — progressing — level 1 thru/or K — it is alike, respectively, 
and the center of gravity of the center of gravity of the pixel which belongs is 
searched for as a center of gravity of the whole pixel which constitutes an attention 
frame, and carries out a return. 

[0062] namely, — step S20 — level 1 thru/or K — it is alike, respectively, and the 
weighting average which makes weight the frequency f 1 thru/ or fK is calculated about 
the center of gravity of the pixel which belongs, and it is outputted as a center of 
gravity of the whole pixel where the weighting average constitutes an attention frame. 
[0063] Next, with reference to the flow chart of drawing 10 , the processing 
(center-of-gravity calculation processing of the motion detection range) which the 
center-of-gravity calculation section 21 performs is explained in full detail in step S4 
of drawing 7 . 

[0064] First, the motion detection range is set up in step S31. That is, at step S31, the 
range where the frame in front of one attention frame was written in is detected from 
the are recording image memorized in the are recording image memory 24. 
Furthermore, at step S31, the detected range can extend only the predetermined 
number of pixels for example, in the direction of four directions, respectively, and the 
range which was able to be extended is set up as motion detection range. 
[0065] And it progresses to step S32, and to Variables X or Y, the number of pixels 
beside the motion detection range or the vertical number of pixels is set, respectively, 
and progresses to step S33. 

[0066] Then, in step S33 thru/or S41, step S12 of drawing 8 thru/or the respectively 
same processing as the case in S20 are performed, and thereby, the center of gravity 
of the whole pixel which constitutes the motion detection range is searched for, and 
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carries out a return. 

[0067] In the camera motion detecting element 12, the predetermined range 
containing the frame in front of one of the attention frames in an are recording image 
is set up as motion detection range. As mentioned above, the center of gravity of the 
motion detection range, Since the center of gravity of an attention frame is computed, 
and it asks for the camera motion vector of an attention frame based on those 
centers of gravity and was made to repeat writing an attention frame in an are 
recording image based on the camera motion vector As compared with the case 
where the so-called block matching is performed, it can ask for a camera motion 
vector simply. 

[0068] In addition, although the center of gravity of an attention frame is searched for 
and the center of gravity of the motion detection range was searched for in step S4 
after that in step S3 with the gestalt of operation of drawing 7 , you may ask for 
whichever first and the center of gravity of an attention frame and the center of 
gravity of the motion detection range can also be made to ask simultaneously. 
[0069] Next, although it asked for each pixel which constitutes the attention frame for 
the center of gravity of an attention frame with the gestalt of operation of drawing 8 
by classifying into either of some level according to the pixel value (suitably 
henceforth a level classification) In addition to this, the center of gravity of an 
attention frame for example, each pixel which constitutes the attention frame It is also 
possible to carry out based on some pixels (for self to be included and for it not to be 
necessary to include) around it by classifying into either of some classes (suitably 
henceforth a class classification). 

[0070] Here, a class classification is explained briefly. A now, for example, attention, 
pixel shall constitute the tap (suitably henceforth a class tap) used for the class 
classification of an attention pixel from the 4 pixels which adjoins vertically and 
horizontally, respectively, and own a total of 5 pixels of an attention pixel. If a pixel 
value shall be expressed by 1 bit in this case (it becomes the value of either 0 or 1), an 
attention pixel can be classified into the pattern of 32 (= (21) 5) according to the pixel 
value which 5 pixels of the class tap constituted about that attention pixel can take. It 
is such a pattern part injury class classification, and an attention pixel will be 
classified into either of the classes of 32 when it is now. 

[0071] In addition, if the class tap which becomes by 5 pixels as mentioned above is 
constituted and a class classification is generally performed when 8 bits is assigned to 
the pixel although about 8 bits is assigned to a pixel, the number of classes will turn 
into a huge number called (28) 5. 
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[0072] Then, to the class tap constituted about an attention pixel, L bit ADRC 
(Adaptive Dynamic Range Coding) processing is carried out, and a class classification 
can be performed based on the class tap after the ADRC processing. 
[0073] Here, in L bit ADRC processing, for example, Maximum MAX and the minimum 
value MIN of a pixel value of a pixel which constitute a class tap are detected, 
DR=MAX-MIN is used as the local dynamic range of a set, and the pixel which 
constitutes a class tap is re-quantized by L bits based on this dynamic range DR. That 
is, out of the pixel value of the pixel which constitutes a class tap, the minimum value 
MIN is subtracted and the division (quantization) of the subtraction value is done by 
DR/2L. Therefore, when L bit ADRC processing of the class tap is carried out, as 
compared with the case where a class classification is performed without carrying out 
ADRC processing of the class tap, the number of classes can be decreased by the 
pixel value of each pixel which constitutes the class tap being made into L bits, and 
considering as a value smaller than the number of bits to which L was assigned by the 
pixel. 

[0074] In addition, moreover, when the pixel of the upper bed of a frame, a soffit, a left 
end, or a right end turns into an attention pixel, since the pixel which adjoins the left or 
the right the bottom does not exist, it shall constitute a class tap, assuming it to be 
that to which the same frame turns up and exists in a frame upside, the bottom, 
left-hand side, or right-hand side in this case. 

[0075] Next, by carrying out the class classification of each pixel which constitutes an 
attention frame with reference to the flow chart of drawing 1 1 based on the class tap 
about the pixel explains the processing (center-of-gravity calculation processing of 
an attention frame) performed in the center-of-gravity calculation section 21 in the 
case of searching for the center of gravity of an attention frame. 

[0076] In this case, in step S51 thru/or S57, step S1 1 of drawing 8 thru/or the 
respectively same processing as the case in S17 are performed. And the pixel p (x y) 
in a coordinate (x y) is made into an attention pixel, and a class tap is constituted from 
step S58 corresponding to step S18 of drawing 8 by the attention pixel. And based on 
the class tap, the class classification of the attention pixel is carried out at either of 
the K classes, and the class classification result is registered into a class table. 
[0077] In the gestalt of operation of drawing 1 1 namely, the centei—of-gravity 
calculation section 21 In the memory to build in, for example, the frequency fk of the 

pixel which belongs to the class k about each class k (k= 1, 2 K) as shown in 

drawing 12 , When the class table which matched integrated value sigmaxk of the 
x-coordinate of the pixel belonging to Class k and integrated value sigmayk of a 
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y-coordinate is memorized, for example, an attention pixel belongs to Class k While 
only 1 increments the frequency fk about the class k in a class table, the x-coordinate 
or y-coordinate of an attention pixel is added to integrated value sigmaxk of an 
x-coordinate, or integrated value sigmayk of a y-coordinate, respectively. 
[0078] In addition, a class table is cleared by 0 whenever processing according to the 
flow chart of drawing 1 1 is started like a level table. 

[0079] The same processing is repeated by step S56 return and the following after 
processing of step S58. 

[0080] On the other hand, when judged with Variable y not being under Y in step S54, 
it progresses to step S59 and the center of gravity of the pixel belonging to each class 
of a class table is searched for. That is, the coordinate (sigmaxk/fk, sigmayk/fk) as 
.which the division of integrated value sigmaxk of the x-coordinate in each class k of a 
class table or each integrated value sigmayk of a y-coordinate is done in the 
frequency fk, and it is expressed in step S59 with the division value is searched for as 
a center of gravity of the pixel belonging to each class k. 

[0081] and the step S60 — progressing — a class 1 thru/or K — it is alike, 
respectively, and the center of gravity of the center of gravity of the pixel which 
belongs is searched for as a center of gravity of the whole pixel which constitutes an 
attention frame, and carries out a return. 

[0082] namely, — step S60 — a class 1 thru/or K — it is alike, respectively, and the 
weighting average which makes weight the frequency f 1 thru/or fK is calculated about 
the center of gravity of the pixel which belongs, and it is outputted as a center of 
gravity of the whole pixel where the weighting average constitutes an attention frame. 
[0083] When searching for the center of gravity of an attention frame by carrying out 
the class classification of each pixel which constitutes an attention frame here based 
on the class tap about the pixel, it is desirable to make it ask by the center of gravity 
of the motion detection range also constituting a class tap about each pixel which 
constitutes the motion detection range, and performing a class classification based on 
the class tap. 

[0084] In addition, in searching for the center of gravity of the motion detection range 
by performing a class classification, in the processing which showed the flow chart of 
drawing 1 1 , it changes to processing of the step S51, and since it is the same as that 
of the case where processed steps S31 and S32 shown in the flow chart of drawing 
10 , and also it is shown in drawing 1 1 , it omits the explanation. 

[0085] Next, although the center of gravity of an attention frame was searched for 
with the gestalt of operation of drawing 1 1 using the center of gravity of the pixel 
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belonging to all the classes that carry out the class classification of each pixel which 
constitutes an attention frame based on the class tap about the pixel, and are 
obtained as a result, the center of gravity of an attention frame can be made to ask 
only using the center of gravity of the pixel which belongs to the specific class 
obtained as a result of for example, a class classification in addition to this. 
[0086] That is, the center of gravity of an attention frame can be asked for the class 
to which the pixel (suitably henceforth an edge pixel) of the part which is an edge 
belongs as a specific class only using the center of gravity of the pixel (edge pixel) 
belonging to the specific class (suitably henceforth an edge class). 
[0087] Then, when asking with reference to the flow chart of drawing 1 3 only using the 
center of gravity of the edge pixel which belongs to an edge class among the class 
classification results of each pixel which constitutes an attention frame, the 
processing (center-of-gravity calculation processing of an attention frame) performed 
in the center-of-gravity calculation section 21 is explained. 

[0088] Also in this case, step S11 of drawing 8 thru/or the respectively same 
processing as the case in S17 are performed in step S71 thru/or S77. And in step S77, 
when judged with Variable x being under X, it progresses to step S78, the pixel p (x y) 
in a coordinate (x y) is made into an attention pixel, and the class classification of the 
attention pixel is carried out like the case in step S58 of drawing 1 1 . 
[0089] Then, it progresses to step S79 and it is judged [ the class obtained as a result 
of the class classification in step S78 ] for it being an edge class, i.e., an attention pixel, 
whether it is an edge pixel. 

[0090] Here, an attention pixel is classified into either of the 1024 (= (22) 5) classes 
when performing a class classification, after carrying out 2-bit ADRC processing of 
the class tap which consists of 5 pixels which was mentioned above, for example. By 
the way, when 2-bit ADRC processing of the class tap is carried out, the pixel value of 
the pixel which constitutes the class tap turns into a value of the 00B, 01 B, 10B, or 
the 1 1 B (B expresses that the value arranged before that is a binary number). 
Therefore, as mentioned above, when a class tap consists of 4 pixels [ an attention 
pixel and / which adjoins the four directions of the attention pixel, respectively ] a 
total of 5 pixels, it sets. While the time of the pixel value of 4 pixels which adjoins 
vertically and horizontally, respectively being except 00B while the pixel value of an 
attention pixel is 00B which is the minimum value, and the pixel value of an attention 
pixel are 1 1 B which is maximum When the pixel value of 4 pixels which adjoins 
vertically and horizontally, respectively is except 1 1 B, it is thought that the attention 
pixel is an edge pixel. 
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[0091] That is, as shown in drawing 14 (A), when the pixel value of an attention pixel is 
00B and the pixel value of 4 pixels which adjoins vertically and horizontally, 
respectively is 01 B, 10B, or the 1 1B, so to speak, the pixel value is a trough (concave) 
in the attention pixel. Moreover, as shown in drawing 14 (B), when the pixel value of an 
attention pixel is 1 1 B and the pixel value of 4 pixels which adjoins vertically and 
horizontally, respectively is 00B, 01 B, or the 10B, so to speak, the pixel value is a 
crest (convex) in the attention pixel. Therefore, the attention pixel is an edge pixel 
when shown in drawing 14 . 

[0092] In addition, when the pixel value of an attention pixel is 00B and the pixel value 
of 4 pixels which adjoins vertically and horizontally, respectively is 01 B, 10B, or the 
11B When the pixel value of a **** and an attention pixel is 1 1 B 81 (= 3x3x3x3) 
passages and the pixel value of 4 pixels which adjoins vertically and horizontally, 
respectively is 00B, 01 B, or the 10B, there are 81 kinds similarly. Therefore, there is a 
class (edge class) to which an edge pixel belongs 162 (= 81+81) passages. 
[0093] When it is judged with an attention pixel not being an edge pixel by drawing 13 
in return and step S79 (i.e., when the classes of an attention pixel are not any of the 
above-mentioned edge classes which have 1 62 kinds, either), it returns to step S76. 
[0094] Moreover, in step S79, when judged with an attention pixel being an edge pixel 
(i.e., when the class of an attention pixel is either of the above-mentioned edge 
classes which have 162 kinds), it progresses to step S80 and the class classification 
result of an attention pixel is registered into a class table. That is, at step S80, while 
the increment of the frequency fk about the class k of an attention pixel of a class 
table as shown in drawing 1 2 is carried out only for 1, the x-coordinate or 
y-coordinate of an attention pixel is added to integrated value sigmaxk of an 
x-coordinate, or integrated value sigmayk of a y-coordinate, respectively. 
[0095] In addition, in performing a class classification and registering only an edge 
class into a class table only about the attention pixel belonging to an edge class with 
the gestalt of operation of drawing 13 , for example after carrying out 2— bit ADRC 
processing of the class tap which consists of 5 pixels as mentioned above since 
registration to a class table is performed, the number K of classes of a class table is 
set to 162 which is the number of edge classes mentioned above. 
[0096] Since it is classified into either of the 1024 classes as the attention pixel was 
mentioned above when performing a class classification and registering all classes into 
a class table on the other hand after carrying out 2-bit ADRC processing of the class 
tap which consists of 5 pixels, the number K of classes of a class table is set to 1024. 
[0097] Therefore, in the gestalt of operation of drawing 1 3 , magnitude (capacity) of a 
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class table can be made small as compared with the case in the gestalt of operation of 
drawing 1 1 . 

[0098] The same processing is repeated by step S76 return and the following after 
processing of step S80. 

[0099] On the other hand, when judged with Variable y not being under Y in step S74, 
it progresses to step S81 and the center of gravity of the pixel belonging to each edge 
class of a class table is searched for. That is, the coordinate (sigmaxk/fk, sigmayk/fk) 
as which the division of integrated value sigmaxk of the x-coordinate in each edge 
class k of a class table or each integrated value sigmayk of a y-coordinate is done in 
the frequency fk, and it is expressed in step S81 with the division value is searched for 
as a center of gravity of the pixel belonging to each edge class k. 
[0100] and the step S82 — progressing — the edge class 1 thru/or K — it is alike, 
respectively, and the center of gravity of the center of gravity of the pixel which 
belongs is searched for as a center of gravity of the whole pixel which constitutes an 
attention frame, and carries out a return. 

[0101] namely, — step S82 — the edge class 1 thru/or K — it is alike, respectively, 
and the weighting average which makes weight the frequency f1 thru/or fK is 
calculated about the center of gravity of the pixel which belongs, and it is outputted as 
a center of gravity of the whole pixel where the weighting average constitutes an 
attention frame. 

[0102] When searching for the center of gravity of an attention frame here only using 
what belongs to an edge class among the pixels which constitute an attention frame, it 
is desirable to also search for the center of gravity of the motion detection range only 
using what belongs to an edge class among the pixels which constitute the motion 
detection range. 

[0103] In addition, in searching for the center of gravity of the motion detection range 
only using the pixel belonging to an edge class, in the processing shown in the flow 
chart of drawing 1 3 , it changes to processing of the step S7 1 , and since it is the same 
as that of the case where processed steps S31 and S32 shown in the flow chart of 
drawing 10 , and also it is shown in drawing 13 , it omits the explanation. 
[0104] Next, when an attention frame is what does not have a camera motion to the 
frame in front of one of them, the camera motion vector of an attention frame should 
become equal to the camera motion vector of the frame in front of one of them. 
However, as it mentioned above, when asking for the camera motion vector of a frame, 
when a motion of a foreground influences, even if there is no camera motion of an 
attention frame, a different thing from the camera motion vector of the frame in front 



JP11258472 



of one of them may be found as the camera motion vector. 

[0105] so, to the camera motion detecting element 12 shown in drawing 5 Form the 
motion-less judgment section 25 which judges whether an attention frame is what 
does not have a camera motion to the frame in front of one of them as a dotted line 
shows to this drawing, and it sets to the vector detecting element 22 further. In being 
a thing without a camera motion of an attention frame Output as a camera motion 
vector of an attention frame, and when an attention frame is a thing with a camera 
motion, the camera motion vector of the frame in front of one of them As it mentioned 
above, it can ask for the camera motion vector of an attention frame from the center 
of gravity of an attention frame and the motion detection range. 

[0106] By doing in this way, when it is a thing without a camera motion of an attention 
frame, the camera motion vector of an attention frame can be made equal to the 
camera motion vector of the frame in front of one of them. 

[0107] Then, with reference to the flow chart of drawing 1 5 , the processing 
(motion-less judgment processing) which judges whether it is the thing without a 
camera motion of an attention frame performed in the motion-less judgment section 
25 is explained. 

[0108] In motion-less judgment processing, step S71 of drawing 13 thru/or the 
respectively same processing as the case in S79 are performed in step S91 thru/or 
S99. 

[0109] And in step S99, when judged with an attention pixel being an edge pixel, it 
progresses to step S100 and it is judged whether it is equal to pixel value p' (x y) of 
the pixel which has the pixel value p of an attention pixel (x y) in the same location in 
front of one of them. 

[01 10] Here, p (x y) in step S100 shall include the case where p (x y) is [ |p(x y)-p'(x y) 
| ] less than a predetermined minute value when almost equal to p' (x y) as it is equal 
to p' (x y). 

[01 1 1] In step S100, when judged with it not being equal to pixel value p' (x y) of the 
pixel which has the pixel value p of an attention pixel (x y) in the same location in front 
of one of them, step S101 is skipped and it returns to step S96. 

[01 1 2] In step S100 moreover, the pixel value p of an attention pixel (x y) When judged 
with it being equal to pixel value p' (x y) of the pixel in the same location in front of one 
of them, That is, when equal to pixel value p' (x y) of the pixel before [ one ] the pixel 
value p of the attention pixel which is an edge pixel (x y) is in the same location 
spatially, it progresses to step S101, and the increment only of 1 is carried out and 
Variable c returns to step S96. 
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[0113] Here, Variable c is cleared by zero before motion-less judgment processing of 
drawing 15 is started. 

[01 14] Then, in step S94, when judged with Variable y not being under Y (i.e., when it 
processes by making into an attention pixel all the pixels that constitute an attention 
frame), it progresses to step S102 and it is judged whether Variable c is beyond the 
predetermined threshold th. When judged with Variable c being beyond the 
predetermined threshold th in step S102, Namely, the pixel used as a pixel value 
[ being the same as that of the pixel which is in the edge pixel of an attention frame in 
the same location of one frame ago (almost the same) ] When it exists beyond the 
threshold th, it progresses to step S103, and as a judgment result of whether an 
attention frame is what does not have a camera motion to the frame in front of one of 
them, the message of a purport without a motion is outputted to the vector detecting 
element 22, and ends motion-less judgment processing. 

[0115] Moreover, when judged with Variable c not being beyond the predetermined 
threshold th in step S102, Namely, the pixel used as a pixel value [ being the same as 
that of the pixel which is in the edge pixel of an attention frame in the same location of 
one frame ago (almost the same) ] When it does not exist beyond the threshold th, 
progress to step S104 and an attention frame receives the frame in front of one of 
them. As a judgment result of whether it is a thing without a camera motion, the 
message of a purport with a motion is outputted to the vector detecting element 22, 
and ends motion-less judgment processing. 

[01 1 6] In addition, motion-less judgment processing of drawing 15 is performed in 
advance of the processing which computes the center of gravity of the attention 
frame in the center-of-gravity calculation section 21, and the motion detection range, 
and is further performed for the frame after the 2nd frame. 

[0117] Next, drawing 16 shows the example of a configuration of the background 
extract section 14 of drawing 2 . 

[0118] The camera motion vector v1 of the 1st frame thru/or the Nth frame as a 
series of images memorized by the camera motion vector storage section 13 ( drawing 
2 ) thru/or vN are supplied to the existence range detecting element 31. The 
existence range detecting element 31 is in the condition which performed alignment of 
the background of the 1st thru/or the Nth frame, and detects the field (existence 
range) of the minimum rectangle where the 1st thru/or image of the Nth frame exists 
in standard coordinates. 

[01 19] That is, based on the camera motion vector v1 of the 1st frame thru/or the 
Nth frame thru/or vN, the existence range detecting element 31 is in the condition 
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which performed the alignment about the 1 st frame thru/or the Nth frame supposing 
the condition of having performed alignment of the background, and detects the 
existence range which is the field of the minimum rectangle where the 1st thru/or the 
pixel of the Nth frame exist. Furthermore, also in the standard coordinates of the 
existence range, the coordinate (Xmin, Ymin) of the top-most vertices of most the 
upper left and the coordinate (Xmax, Ymax) of the top-most vertices of most the 
lower right are searched for, and the existence range detecting element 31 is supplied 
to the read-out section 32 and the write-in section 35. 

[0120] In addition, the condition of having performed alignment of the background of 
the 1st frame thru/or the Nth frame can be assumed in standard coordinates by 
arranging the 1st frame thru/or the Nth frame so that the top-most vertices at the 
upper left of a frame may be located in the coordinate shown by each camera motion 
vector v1 thru/or vN, as shown in drawing 17 . 

[0121] The read-out section 32 detects and reads the pixel which is in the same 
location spatially, where alignment of the background of the 1st frame thru/or the Nth 
frame is performed among the pixels which constitute the 1 st frame memorized by the 
are recording section 1 1 ( drawing 2 ) thru/or the Nth frame, and it supplies it to the 
frequency count area 33 and the write-in section 35. 

[0122] namely, — the read-out section 32 — the coordinate (Xmin, Ymin) from the 
existence range detecting element 21 — and (Xmax, Ymax) it being supplied and also 
The camera motion vector v1 memorized by the camera motion vector storage 
section 13 ( drawing 2 ) thru/or vN are also supplied. The read-out section 32 First, 
based on the camera motion vector v1 of the 1st frame thru/or the Nth frame thru/or 
vN, the condition of having performed alignment of the background about the 1st 
frame thru/or the Nth frame is assumed like the existence range detecting element 31 . 
Furthermore, the read-out section 32 carries out the sequential scan of the 
coordinate (Xmin, Ymin) supplied from the existence range detecting element 21 of 
standard coordinates, and (Xmax, Ymax) the coordinate of existence within the limits 
specified, and as shown in drawing 18 , it detects and reads the pixel of the 1 st frame 
thru/or the Nth frame in the condition of having performed alignment of a background 
in each coordinate (x y). 

[0123] In addition, detection of the pixel of the n-th frame located in the coordinate (x 
y) of standard coordinates in the condition of having performed alignment of the 
background of the 1 st thru/or the Nth frame When the local system of coordinates 
(suitably henceforth local system of coordinates) which made the zero the pixel at the 
upper left of [ the ] the n-th frame are considered so to speak, it can carry out by 
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detecting the pixel which is in the coordinate which subtracted the camera motion 
vector vn from a coordinate (x y). 

[0124] The inside of the pixel to which the frequency count area 33 is supplied from 
the read-out section 32 and which constitutes the 1 st thru/or the Nth frame. The 
pixel which is in the same location spatially where alignment of the background is 
performed every [ of (calling it hereafter the same location pixel in an alignment 
condition suitably) ] set — for example, the frequency of the pixel belonging to each 
level of the pixel value mentioned above — counting — carrying out — the counting 
— based on a result, registration to the frequency table memorized by the frequency 
table storage section 34 of the latter part is performed. 

[01 25] The frequency table storage section 34 memorizes a frequency table as shown 
in drawing 19 . That is, the frequency table storage section 34 has memorized the 
frequency table for matching and registering each level of the pixel value of the same 
location pixel in an alignment condition, and the rate to the number of pixels of the 
same location pixel in the alignment condition of the frequency of the pixel belonging 
to the level about each coordinate (x y) of existence within the limits. 
[0126] Here, in the set of the same location pixel in the alignment condition in a 
certain location (coordinate) of existence within the limits, the level of the large pixel 
value to the m-th of frequency is called m-th frequency level. 

[01 27] With the gestalt of operation of drawing 19 , M level and the rate of frequency 
from the 1st frequency level to the Mth frequency level are registered into a 
frequency table, therefore — here — the frequency count area 33 — frequency — 
the level 1st after M+ — counting of frequency — a result is not registered into a 
frequency table but is canceled, however, counting of frequency — a result can also 
be made to register with a frequency table about all level 

[0128] In addition, as mentioned above, when the range of the value which can be 
taken as a pixel value is divided into K level, above-mentioned M becomes a value 
below K. 

[0129] The write-in section 35 writes the background pixel which constitutes a whole 
background based on the frequency table memorized by the frequency table storage 
section 34 and the pixel supplied from the read-out section 32 in each address 
equivalent to existence within the limits supplied from the existence range detecting 
element 31 of the background memory 15 ( drawing 2 ). Furthermore, the write-in 
section 35 also performs the writing of the background flag to the background flag 
memory 36. 

[0130] The background flag memory 36 memorizes the background flag with which it 
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expresses whether the background pixel is written in about each pixel of existence 
within the limits. That is, the write-in section 35 writes a background flag in the 
address of the background flag memory 36 corresponding to the address, when a 
background pixel is written in the address with the background memory 15. Here, a 
background flag shall presuppose that it is a 1 —bit flag, the background flag 
corresponding to the address with which the background pixel is written in shall be set 
to 1, and the background flag corresponding to the address which is not written in yet 
shall be set to 0. 

[0131] Next, with reference to the flow chart of drawing 20 , background extract 
processing in which a whole background is extracted is explained from the 1st frame 
performed in the background extract section 14 of drawing 16 thru/or the Nth frame. 
[0132] First, in step S1 11, as the existence range detecting element 31 reads a 
camera motion vector from the camera motion vector storage section 13 and 
explained it by drawing 1 7 , it detects the existence range. And the coordinate (Xmin, 
Ymin) of the point of the upper left in the standard coordinates of the existence range 
and the coordinate (Xmax, Ymax) of a lower right point are supplied to the read-out 
section 32 and the write-in section 35 as information for pinpointing the existence 
range. 

[0133] Ymin-1 as initial value is set to the coordinate (Xmin, Ymin) for pinpointing the 
existence range, and (Xmax, Ymax) the variable y for scanning the existence range to 
y shaft orientations of standard coordinates in step S112, if it receives, it progresses 
to step S113, only 1 increments the variable y, and the read-out section 31 
progresses to step S114. At step S114, when judged with it being judged whether 
Variable y is below Ymax and being below Ymax, it progresses to step S115. At step 
S115, the read-out section 31 sets Xmin-1 as initial value to the variable x for 
scanning the existence range in the direction of a x axis of standard coordinates, 
progresses to step S116, and only 1 increments the variable x and it progresses to 
step S1 1 7. At step S1 1 7, when judged with it being judged whether Variable x is below 
Xmax and not being below Xmax, the same processing is repeated by step S1 1 3 return 
and the following. 

[0134] Moreover, in step S117, when judged with Variable x being below Xmax, it 
progresses to step S1 18, and 0 as initial value is set to the variable n for counting the 
frame number of the 1 st frame thru/or the Nth frame as a series of images memorized 
by the are recording section 1 1 ( drawing 2 ), and it progresses to step S1 19. At step 
S1 1 9, it is judged whether Variable n is below N that is the frame number of a series of 
images with which the increment only of 1 was carried out, it progressed to step S120, 
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and Variable n was memorized by the are recording section 1 1 ( drawing 2 ). 

[0135] In step S120, when judged with Variable n being below N, it progresses to step 

5121, and where alignment of the background of the 1st frame thru/or the Nth frame 
is performed, in the read-out section 32, reading appearance of the pixel of the n-th 
frame in the location of the coordinate (x y) of standard coordinates is carried out 
from the are recording section 1 1 ( drawing 2 ). That is, the read-out section 32 is the 
coordinate (x y) which subtracted the camera motion vector vn from the coordinate (x 
y) in the local system of coordinates which made the zero the pixel at the upper left of 
the n-th frame. - The pixel in vn is read from the are recording section 1 1 ( drawing 2 ). 
[0136] In addition, while expressing the x-coordinate or y-coordinate of the camera 
motion vector vn as xv#n or yv#n, respectively If the side of one frame or the vertical 
number of pixels is expressing X or Y, respectively, although the read-out section 32 
will read the pixel located in the coordinate (x-xv#n, y-yv#n) in the local system of 
coordinates about the n-th frame In this case, the pixel of the n-th frame does not 
exist in the coordinate (x-xv#n, y-yv#n) of 0 <=x-xv#n<X and 0 <=y-yv#n<Y which 
can be set out of range. Therefore, read-out of the pixel of the n-th frame in step 
S121 is performed, only when x~xv#n is within the limits of 0 <=x~xv#n<X and y-yv#n 
is within the limits of 0 <=y-yv#n<Y. 

[0137] In step S121, if the pixel of the n-th frame is read, the read-out section 32 will 
supply the pixel to the frequency count area 33 and the write-in section 35, and will 
return from the are recording section 11 ( drawing 2 ) to step S119. And until it is 
judged with Variable n not being below N in step S120 Step S1 19 thru/or processing 
of S121 are repeated, and by this, where alignment of the background of the 1st frame 
thru/or the Nth frame is performed The pixel (the same location pixel in an alignment 
condition) of the 1st frame in the location of the coordinate (x y) of standard 
coordinates thru/or the Nth frame is supplied to the frequency count area 33 and the 
write-in section 35. However, as mentioned above, depending on a frame, a pixel may 
not exist in the coordinate (x y) of standard coordinates, and the pixel of that frame is 
not contained in the pixel supplied to the frequency count area 33 and the write-in 
section 35 in this case. 

[0138] If judged with Variable n not being below N in step S120, it will progress to step 

5122. Then, the frequency count area 33 The pixel value carries out the level 
classification of each pixel of the 1st thru/or the Nth frame in the condition of having 
performed alignment of a background in the location of the coordinate (x y) of 
standard coordinates supplied from the read-out section 32 by whether it belongs to 
level 1 thru/or which range of the K. Furthermore, the frequency count area 33 
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carries out counting of the frequency of the pixel belonging to each level, and asks for 
the rate (rate to the total of the pixel belonging to each level) of the frequency. 
[0139] And it progresses to step S123, and the frequency count area 33 is registered 
into the column (line) of a coordinate (x y) in a frequency table as showed the rate of 
the frequency of a pixel that the rate of frequency belongs to the level (the 1st 
frequency level thru/or the Mth frequency level) and each of its level from the 1 st 
place to the Mth place to drawing 19 of the frequency table storage section 34, and 
returns to step S1 1 6. 

[0140] When it is judged with Variable y not being below Ymax in step S114 on the 
other hand, When all the coordinates of existence within the limits are processed, it 
progresses to step S124. Namely, the write-in section 35 The coordinate which the 
rate of the pixel belonging to the 1st frequency level has become in the frequency 
table memorized by the frequency table storage section 34 beyond the predetermined 
value is detected. Background pixel extract processing which writes in the pixel value 
corresponding to the 1st frequency level as a pixel value of a background pixel is 
performed to the address of the background memory 15 corresponding to the 
coordinate, and it goes to it at step S125. At step S125, about the coordinate of 
existence within the limits in which a pixel value was not written in background pixel 
extract processing of step S124, the write-in section 35 performs background escape 
processing which writes in the pixel value as a background pixel, and ends background 
extract processing. 

[0141] Next, with reference to the flow chart of drawing 21 , the background pixel 
extract processing which writes in in step S124 of drawing 20 , and the section 35 
performs is explained. 

[0142] In background pixel extract processing, in step S131 thru/or S136, step S112 
of drawing 20 thru/or the respectively same processing as the case in 117 are 
performed, and it sets to step S136. When judged with Variable x being below Xmax, it 
is judged by progressing to step S37 and referring to a frequency table whether the 
rate of the 1 st frequency level to a coordinate (x y) is beyond the predetermined 
threshold Lth. 

[0143] When judged with the rate of the 1st frequency level to a coordinate (x y) not 
being beyond the predetermined threshold Lth in step S137, That is, when the rate of 
the frequency of the pixel value with most frequency of the pixel located in a 
coordinate (x y) among the pixels which constitute the 1st frame in the condition of 
having performed alignment of a background thru/or the Nth frame is not high, steps 
S138 and S139 are skipped, and it returns to step S135. 
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[0144] Therefore, the pixel value of a background pixel is not written in the address of 
the background memory 1 5 ( drawing 2 ) corresponding to the coordinate (x y) in the 
existence range in this case. 

[0145] When it is judged with the rate of the 1st frequency level to a coordinate (x y) 
being beyond the predetermined threshold Lth in step S1 37 on the other hand, Namely, 
the inside of the pixel which constitutes the 1 st frame in the condition of having 
performed alignment of a background thru/or the Nth frame, When the rate of the 
frequency of the pixel value with most frequency of the pixel located in a coordinate (x 
y) is high, it progresses to step S138 and the write-in section 35 writes a pixel value 
with the high rate of the frequency in the background memory 1 5 as a pixel value of 
the background pixel located in a coordinate (x y). 

[0146] That is, the write-in section 35 extracts what belongs to the 1st frequency 
9 level among the pixels located in the coordinate (x y) of the 1 st frame in the condition 
of having performed alignment of a background thru/or the Nth frame from the pixel 
supplied from the read-out section 32, for example, calculates the average. And the 
write-in section 35 writes the average in the background memory 1 5 as a pixel value 
of the background pixel located in a coordinate (x y). 

[0147] After processing of step S138, it progresses to step S139, the write-in section 
35 sets to 1 the background flag memorized to the address corresponding to the 
coordinate (x y) of the background flag memory 36 (leave a background flag), and the 
same processing is repeated until it is judged with Variable y not being below Ymax by 
step S135 in step S133 return and the following. 

[0148] And in step S133, if judged with Variable y not being below Ymax, a return will 
be carried out. 

[0149] Here, the storage value of the background flag memory 36 is cleared by 0 when 
background pixel extract processing of drawing 21 is started. 

[0150] Next, in background pixel extract processing in which it explained by drawing 
21 , as mentioned above, when the rate of the frequency of the pixel value with most 
frequency of the pixel located in a coordinate (x y) among the pixels which constitute 
the 1 st frame in the condition of having performed alignment of a background thru/or 
the Nth frame is not high, the pixel value as a background is not written in a 
coordinate (x y). 

[0151] Namely, the inside of the pixel which constitutes the 1st frame in the condition 
of having performed alignment of a background thru/or the Nth frame from 
background pixel extract processing, When the rate of the frequency of the pixel value 
with most frequency of the pixel located in a coordinate (x y) is high, it is written in the 
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background memory 1 5 as a thing with most pixel values of the frequency probable as 
a pixel value of the whole background in a coordinate (x y). therefore, when most pixel 
values of frequency cannot say that it is probable as a pixel value of the whole 
background in a coordinate (x y), here The inside of the pixel which constitutes the 1st 
frame in the condition of having performed alignment of a background thru/or the Nth 
frame, When the rate of the frequency of the pixel value with most frequency of the 
pixel located in a coordinate (x y) is not high, in background pixel extract processing, 
the pixel value of a whole background is not written in the address of the background 
memory 1 5 corresponding to a coordinate (x y). 

[0152] Consequently, by background pixel extract processing, so to speak, the whole 
background which consists of that a pixel value is written in the background memory 
1 5 is in the vermin condition, and needs to bury the vermin part. For this reason, in 
f background extract processing of drawing 20 , after background pixel extract 
processing ( drawing 21 ) is performed at step S124, in step S125, background escape 
processing which extends the background so to speak is performed by fill uping a 
vermin part with a pixel value. 

[01 53] Then, with reference to the flow chart of drawing 22 , the background escape 
processing in step S125 of drawing 20 is explained. 

[0154] In background escape processing, in step S141, Ymin-1 as initial value is set to 
Variable y, and it progresses to step S142, and the increment only of 1 is carried out 
and the variable y progresses to step S143. At step S143, when judged with it being 
judged whether Variable y is below Ymax and being below Ymax, it progresses to step 
S145. At step S145, Xmin-1 as initial value is set to Variable x, and it progresses to 
step S146, and the increment only of 1 is carried out and the variable x progresses to 
step S147. At step S147, when judged with it being judged whether Variable x is below 
Xmax and not being below Xmax, the same processing is repeated by step S142 return 
and the following. 

[0155] Moreover, in step S147, when judged with Variable x being below Xmax, it 
progresses to step S148 and it is judged whether the background flag memorized to 
the address of the background flag memory 36 corresponding to a coordinate (x y) is 0. 
When judged with the background flag memorized to the address of the background 
flag memory 36 corresponding to a coordinate (x y) not being 0 in step S148, That is, 
the background flag is 1, therefore when the pixel value as a whole background is 
already written in the address of the background memory 15 ( drawing 2 ) 
corresponding to a coordinate (x y), the same processing is repeated by step S146 
return and the following. 
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[0156] Moreover, when judged with the background flag memorized to the address of 
the background flag memory 36 corresponding to a coordinate (x y) being 0 in step 
S148, To namely, the address of the background memory 15 ( drawing 2 ) 
corresponding to a coordinate (x y) When the pixel value as a whole background is not 
written in, it still progresses to step S149. Moreover as a surrounding pixel of the pixel 
located in a coordinate (x y), the bottom, it is judged because whether the background 
flag about either of the pixels which adjoin the left, the right, the upper left, the lower 
left, the upper right, and the lower right is 1 refers to the background flag memory 36. 
[0157] When judged with neither of the background flags of the pixel which adjoins the 
pixel located in a coordinate (x y) in step S149 being 1, Namely, a coordinate (x y-1), (x, 
y + 1 ), (x-1 , y), (x+1 , y), When the pixel value as a whole background is written in neither 
of the address of the background memory 15 ( drawing 2 ) corresponding to (x-1, y-1), 
S (x-1 , y+1), (x+1 , y-1), and each (x+1 , y+1) yet, it returns to step S146. 

[01 58] Moreover, when judged with one background flag of the pixels which adjoin the 
pixel located in a coordinate (x y) in step S149 being 1, Namely, a coordinate (x y-1), (x, 
y+1), (x-1, y), (x+1, y), When the pixel value as a whole background is already written in 
either of the addresses of the background memory 15 ( drawing 2 ) corresponding to 
(x-1. y-1), (x-1, y+1), (x+1, y-1), and each (x+1, y+1), it progresses to step S150. 
[0159] Here, that in which the pixel value as a whole background is written among the 
pixels (address of the corresponding background memory 15 ( drawing 2 )) which 
adjoin the pixel located in a coordinate (x y) is hereafter called written in contiguity 
pixel suitably. 

[0160] At step S150, it is judged whether the level of the pixel value of a written in 
contiguity pixel and a continuous pixel value is registered into the frequency table as 
either the 1st to a coordinate (x y) or thru/or the Mth frequency level. Here, a 
continuous pixel value means pixel values (the same pixel value is included) with a 
near value. 

[0161] In step S150, the level of the pixel value of a written in contiguity pixel and a 
continuous pixel value as either the 1st to a coordinate (x y) or thru/or the Mth 
frequency level When judged with registering with the frequency table, constitute the 
1st frame in the condition of having performed alignment of a background thru/or the 
Nth frame. What has the pixel value which follows the pixel value of a written in 
contiguity pixel in the pixel located in a coordinate (x y) the case where it exists more 
than a certain amount of rate — step S151 — progressing — the — Based on the 
pixel value which exists more than a certain amount of rate and which follows the pixel 
value of a written in contiguity pixel, the pixel value as a whole pixel background 
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located in a coordinate (x y) is computed, and it is written in the address with which 
the background memory 1 5 ( drawing 2 ) corresponds. 

[0162] Namely, supposing the level of the pixel value of a written in contiguity pixel 
and a continuous pixel value was registered into the frequency table as the m-th 
frequency level of the 1st to a coordinate (x y) thru/or the Mth frequency level 
Constitute the 1st frame in the condition of having performed alignment of a 
background thru/or the Nth frame from a step S151. The average of the pixel value 
belonging to the m-th frequency level of the pixels located in a coordinate (x y) is 
computed, and the average is written in the address with which the background 
memory 15 ( drawing 2 ) corresponds as a pixel value as a whole pixel background 
located in a coordinate (x y). 

[0163] As mentioned above, here in background pixel extract processing The inside of 
the pixel which constitutes the 1st frame in the condition of having performed 
alignment of a background thru/or the Nth frame, Although it is written in the 
background memory 15 as a thing with most pixel values of the frequency probable as 
a pixel value of the whole background in a coordinate (x y) when the rate of the 
frequency of the pixel value with most frequency located in a coordinate (x y) is high 
For example, a certain scenery is made into a background, according to this 
background pixel extract processing When a series of images which become with the 
N frame which a certain body moves as a foreground are considered Since the rate of 
the frequency of the pixel value currently continued and displayed on almost all the 
frames about the pixel as which most N frames are covered and the background is 
displayed becomes high, without being hidden by the foreground, the pixel value is 
written in as a pixel value of a background. 

[0164] However, the pixel (suitably henceforth a medium pixel) as which a foreground 
is displayed when a foreground moves, or a background is displayed Since it becomes 
the pixel value which constitutes a foreground or becomes the pixel value which 
constitutes a background, since the pixel value to which the rate of frequency 
becomes high does not exist, about a medium pixel, the writing of a pixel value is not 
performed by background pixel extract processing. 

[01 65] Although what is necessary is just to write in the pixel value of the frame as 
which the background is displayed there about a medium pixel in order to acquire a 
whole background, it is difficult to specify the frame as which the background is 
displayed on the medium pixel. Then, if there are some in which the pixel value as a 
whole background is already written and the pixel value and a near pixel value may be 
displayed as a pixel value of a medium pixel in background escape processing into the 
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pixel which adjoins a medium pixel It is written in the background memory 15 ( drawing 
2 ) noting that a pixel value with the thing [ having been displayed ] is probable as a 
pixel value when a background is displayed on a medium pixel. 

[0166] Therefore, according to background escape processing, it becomes possible to 
write in the pixel value of the background extremely displayed in one of them only in 
one certain frame also about the pixel (medium pixel) as which the background was 
displayed. 

[0167] On the other hand in step S150, the level of the pixel value of a written in 
contiguity pixel and a continuous pixel value as either the 1 st to a coordinate (x y) or 
thru/or the Mth frequency level When judged with not registering with a frequency 
table, constitute the 1st frame in the condition of having performed alignment of a 
background thru/or the Nth frame. When what has the pixel value which follows the 
pixel value of a written in contiguity pixel does not exist in the pixel located in a 
coordinate (x y), It progresses to step S152, the pixel value as a whole pixel 
background located in a coordinate (x y) is computed based on the pixel value of a 
written in contiguity pixel, and it is written in the address with which the background 
memory 15 ( drawing 2 ) corresponds. 

[0168] Namely, although some (written in contiguity pixel) in which the pixel value as a 
whole background is already written are in the pixel which adjoins a medium pixel 
When the pixel value and a near pixel value are not displayed as a pixel value of a 
medium pixel In step S1 52, the pixel value (for example, the average when two or more 
written in contiguity pixels exist) of a written in contiguity pixel is written in the 
background memory 15 ( drawing 2 ) as a pixel value as a whole pixel background 
located in a coordinate (x y). 

[0169] After processing of steps S151 and S152, it progresses to step S153, the 
background flag of the address corresponding to the coordinate (x y) of the 
background flag 36 of a pixel with which the pixel value as a whole background was 
written in at steps S151 or S152, i.e., background flag memory, is set to 1 by each 
(built), and the same processing is repeated by step S146 return and the following. 
[0170] On the other hand, when judged with Variable y not being below Ymax in step 
S143, it is judged whether all the background flags to each coordinate of existence 
within the limits which progressed to step S144 and was memorized by the foreground 
flag memory 36 are 1. In step S143, when judged with some which are not 1 being in 
the background flag corresponding to each coordinate of existence within the limits 
(i.e., when some in which the pixel value as a whole background is not written yet are 
in the pixel located in the coordinate of existence within the limits), the same 
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processing is repeated by step S141 return and the following. 

[0171] Moreover, in step S144, when judged with there being nothing that is not 1 into 
the background flag corresponding to each coordinate of existence within the limits 
(i.e., when the pixel value as a whole background is written in all the pixels located at 
the coordinate of existence within the limits), a return is carried out. 
[0172] The pixel in which according to the processing explained by drawing 22 the 
pixel value as a whole background was already written as shown in drawing 23 (in this 
drawing) - When the pixel P (x y) (O mark which attached the slash shows this 
drawing) which adjoins that the mark shows and in which the pixel value is not written 
yet exists The written in contiguity pixel which adjoins the pixel P (x y) (in this 
drawing) The pixel value of Pixels P (x-1, y), P (x-1, y-1), P (x y-1), and P (x+1, y+1), 
the pixel value, the pixel value which has a continuity, etc. are written in as a pixel 
value as a whole pixel P (x y) background, and, thereby, the whole background is 
extended. From this, processing of drawing 22 is called background escape processing. 
[0173] Next, drawing 24 shows the example of a configuration of the foreground 
coding section 1 6 of drawing 2 . 

[0174] The 1st frame as a series of images memorized by the are recording section 1 1 
( drawing 2 ) thru/or the Nth frame, the whole background memorized by the 
background memory 1 5 ( drawing 2 ), and the camera motion vector memorized by the 
camera motion vector storage section 1 3 ( drawing 2 ) are supplied to the foreground 
extract section 41, and the foreground extract section 41 extracts a foreground from 
the 1 st frame thru/or each Nth frame. That is, it is arranging the n-th frame, and the 
foreground extract section 41 performs alignment of a whole background and the n~th 
frame, it is subtracting the pixel of the whole background in the same location from 
each pixel of the n-th frame, and extracts a foreground from the n-th frame so that 
the point at the upper left of the n-th frame may be located in the location where only 
the camera motion vector vn shifted in the standard coordinates of a whole 
background. 

[0175] The foreground storage section 42 memorizes the 1st thru/or the foreground 
of the Nth frame which the foreground extract section 41 extracted. 
[01 76] The camera motion vector memorized by the camera motion vector storage 
section 13 ( drawing 2 ) is supplied to the foreground are recording image 
configuration section 43, and the foreground are recording image configuration section 
43 constitutes a front are recording image and a back are recording image using the 
foreground of the 1st frame thru/or the Nth frame memorized by the foreground 
storage section 42 based on the camera motion vector. Namely, the foreground are 
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recording image configuration section 43 is based on a camera motion vector, as 
shown in drawing 25 . The foreground of the 1 st frame thru/or the Nth frame in the 
condition of having performed alignment of a background is assumed. The back are 
recording image obtained when the foreground of the 1 st frame thru/or the Nth frame 
is seen toward the travelling direction of time amount (image constituted by the 
foreground at which it looked from the past), The front are recording image (image 
constituted by the foreground at which it looked from the future) obtained when it 
goes to the travelling direction and hard flow of time amount is constituted. 
[01 77] In addition, a front are recording image is the sequence of the 1 st frame to the 
Nth frame, and can acquire the pixel value of the foreground of the 1 st frame thru/or 
the Nth frame in the condition of having performed alignment of a background, by 
overwriting memory etc. Similarly, a back are recording image is the sequence of the 
Nth frame to the 1 st frame, and can acquire the pixel value of the foreground of the 
1 st frame thru/or the Nth frame in the condition of having performed alignment of a 
background, by overwriting memory etc. 

[0178] The foreground are recording image storage section 44 memorizes the front 
are recording image and back are recording image which were constituted in the 
foreground are recording image configuration section 43. 

[0179] The study section 45 performs study processing which asks for the prediction 
coefficient for predicting the pixel which constitutes the foreground of the 1st frame 
thru/or each Nth frame from the pixel which constitutes the front are recording image 
memorized by the foreground are recording image storage section 44 and a back are 
recording image, and the error image which was further memorized by the error image 
storage section 49 if needed, and which is mentioned later. 

[0180] The prediction coefficient storage section 46 memorizes the prediction 

coefficient called for by performing study in the study section 45. 

[0181] The adaptation processing section 47 performs adaptation processing which 

predicts the foreground of the 1 st frame thru/or each Nth frame using the pixel which 

constitutes the front are recording image and back are recording image which were 

memorized by the foreground are recording image storage section 44, the prediction 

coefficient memorized by the prediction coefficient storage section 46, and the error 

image further memorized by the error image storage section 49 if needed. 

[0182] Here, the study processing performed in the study section 45 and the 

adaptation processing performed in the adaptation processing section 47 are 

explained. 

[0183] The forecast of the pixel which constitutes the image (here a front are 
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recording image and a back are recording image, and an error image) which exists now 
from adaptation processing, for example, and the pixel which constitutes a desired 
image (here foreground of the 1st frame thru/or each Nth frame) by linear 
combination with a predetermined prediction coefficient is calculated. 
[0184] On the other hand, by study processing, while using a desired image (suitably 
henceforth a request image) as educator data The image (suitably henceforth an 
existence image) which exists when it is going to ask for the request image is used as 
student data. Forecast [ of the pixel value y of the pixel (suitably henceforth a request 
pixel) which constitutes a request image ] E [y] For example, the pixel value x1 of a 
some existence pixel (pixel which constitutes an existence image), x2, and the set of .... 
The prediction coefficient for asking with the primary linearity joint model specified by 
the predetermined prediction coefficients w1 and w2 and the linear combination of ... 
is called for. In this case, forecast E [y] can be expressed with a degree type. 
[0185] 

E[y] =w1x1+w2 x2+ (1) 

[0186] It is [Equation 1] about matrix Y' which becomes by the matrix X which 
becomes by the matrix W which becomes by the set of a prediction coefficient wj, and 
the student data aggregate in order to generalize a formula (1), and the set of forecast 
E[y]. 

Xii X12 •■■ Xu 
x= X21 X22 ■■ X2J 

Xn XI2 ••• Xu 
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If a definition is come out and given, the following observation equations will be 

materialized. 

[0187] 

XW=Y' ... (2) 

Here, the component xij of Matrix X means the j-th student data in the student data 
aggregate (student data aggregate used for prediction of the educator data yi of the 
i-th affair) of the i — th affair, and the component wj of Matrix W expresses the 
prediction coefficient which a product with the j-th student data in the student data 
aggregate calculates. Moreover, yi expresses the educator data of the i-th affair, 
therefore E [yi] expresses the forecast of the educator data of the i-th affair. In 
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addition, y in the left part of a formula (1) omits the suffix i of the component yi of 
Matrix Y, and x1 in the right-hand side of a formula (1), x2, and ... also omit the suffix i 
of the component xij of Matrix X. 

[01 88] And it considers applying a least square method to this observation equation, 
and asking for forecast E [y] near the pixel value y of a request pixel. In this case, it is 
[Equation 2] about the matrix E which becomes by the set of the matrix Y which 
becomes by the set of the true pixel value y of the request pixel used as educator 
data, and the remainder e of forecast E [y] to the pixel value y of a request pixel. 



If a definition is come out and given, the following remainder equations will be 

materialized from an equation (2). 

[0189] 

XW=Y+E ... (3) 

[0190] In this case, the prediction coefficient wj for asking for forecast E [y] near the 
pixel value y of a request pixel is a square error [several 3]. 



It can ask by making it min. 

[0191] Therefore, it will be called an optimum value, when what differentiated the 
above-mentioned square error with the prediction coefficient wj is set to 0, namely, in 
order that the prediction coefficient wj which fills a degree type may ask for forecast 
E [y] near the pixel value y of a request pixel. 



... (4) 

[0193] Then, a degree type is first materialized by differentiating a formula (3) with a 
prediction coefficient wj. 
[0194] 
[Equation 5] 





[0192] 
[Equation 4] 
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... (5) 

[0195] A formula (6) is obtained from a formula (4) and (5). 

[0196] 

[Equation 6] 

i i r 

I e j x j 1 =0, le i xi 2=0.1- ■ -e j x-, j=0 

... (6) 

[01 97] Furthermore, if the relation of the student data xij in the remainder equation of 
an equation (3), a prediction coefficient wj, the educator data yi, and Remainder ei is 
taken into consideration, the following normal equations can be obtained from an 
equation (6). 
[0198] 
[Equation 7] 

( j|xjlXii)Wi + (_|; XjiXj 2 )W2 + --- + ( i | XiiXij)Wj=( I XjiYi) 
( I Xi 2 Xj l)*1+(Z Xi2X i 2)w 2 +... + ( | | Xj 2 Xjj)Wj = ( £ Xjzyj) 

( £ XijXii)Wi-|-(X X iJ Xi2)w 2 + ... + ( i XijXjj)wj=( i Xjjyi) 

i=i i=i i=i i=i 

... (7) 

[0199] Each equation which constitutes the normal equation of an equation (7) is that 
only a certain amount of number prepares the set of the student data xij and the 
educator data yi. Only the same number as several J of the prediction coefficient wj 
for which it should ask can be built, therefore a formula (7) can be asked for the 
optimal prediction coefficient wj by solution Lycium chinense (however, in order to 
solve a formula (7), in a formula (7), the matrix which consists of multipliers concerning 
a prediction coefficient wj needs to be regular). In addition, in solving a formula (7), it is 
possible to sweep out and to, use law (method of elimination of Gauss-Jordan) etc. for 
example. 

[0200] Study processing asks for the optimal prediction coefficient wj as mentioned 
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above, and adaptation processing asks for forecast E [y] near the pixel value y of a 
request pixel by the formula (1) using the prediction coefficient wj. 
[0201] That is, in the study section 45 or the adaptation processing section 47, while 
using as an existence image the front are recording image memorized by the 
foreground are recording image storage section 44 and a back are recording image, 
and the error image further memorized by the error image storage section 49 if 
needed, study processing or adaptation processing is performed, respectively 
considering the foreground of the 1st frame thru/or each Nth frame as a request 
image. 

[0202] In addition, although adaptation processing is not included in an existence 
image, it is the point that the component contained in a request image is reproduced, 
for example, differs from mere interpolation processing. That is, in adaptation 
processing, as long as a formula (1) is seen, it is the same as that of the interpolation 
processing using the so-called interpolation filter, but since [ for which the prediction 
coefficient w equivalent to the tap multiplier of the interpolation filter uses the 
educator data y ] it asks by study so to speak, the component contained in a request 
image is reproducible. From this, adaptation processing can be called processing 
which, so to speak, has a creation (resolution imagination) operation of an image. 
[0203] The error count section 48 reads the foreground of the 1 st frame thru/ or each 
Nth frame from the foreground storage section 42, and calculates the prediction error 
of the forecast of the foreground of the 1 st frame thru/or each Nth frame called for in 
the adaptation processing section 47. Namely, the error count section 48 calculates a 
prediction error for every pixel by subtracting the true value of the pixel value of the 
pixel from the forecast of the pixel which constitutes the foreground of the n-th frame. 
[0204] The error image storage section 49 memorizes the image (suitably henceforth 
an error image) which becomes by the prediction error of the foreground of the 1 st 
frame thru/or the Nth frame called for in the error count section 48. 
[0205] The error judging section 50 calculates for example, the absolute value sum of 
the prediction error as a pixel value of each pixel which constitutes the error image 
memorized by the error image storage section 49, and judges whether the absolute 
value sum has become below a predetermined threshold (following). 
[0206] MUX (multiplexer)51 is based on the judgment result by the error judging 
section 50. To the prediction coefficient and pan which were memorized by the front 
are recording image memorized by the foreground are recording image storage section 
44 and a back are recording image, and the prediction coefficient storage section 46 
The error image memorized by the error image storage section 49 is multiplexed if 
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needed, and the multiplexing data obtained as a result are outputted to a multiplexer 
1 7 ( drawing 2 ) as a coding result of the foreground of the 1 st frame thru/or each Nth 
frame. 

[0207] Next, drawing 26 shows the example of a configuration of the study section 45 
of drawing 24 . 

[0208] The front are recording image memorized by the foreground are recording 
image storage section 44 and a back are recording image (suitably henceforth [ both 
are included and ] a foreground are recording image), and the error image further 
memorized by the error image storage section 49 if needed are supplied to the 
prediction tap configuration section 61. And the prediction tap configuration section 
61 sets as an attention pixel what is going to calculate a forecast among the pixels 
which constitute the foreground of the 1st frame thru/or each Nth frame in the 
condition of having performed alignment of a background to standard coordinates. The 
pixel of the front are recording image in the location near an attention pixel and a 
space target and a back are recording image and the pixel of an error image are 
extracted, and it outputs to the normal equation configuration section 62 as a 
prediction tap used for calculating the forecast of an attention pixel by adaptation 
processing. 

[0209] A prediction tap is supplied from the prediction tap configuration section 61, 
and also the pixel which constitutes the foreground of the 1 st frame memorized by the 
foreground storage section 42 thru/or each Nth frame is supplied to the normal 
equation configuration section 62. And the normal-equation configuration section 62 
performs the add lump for the pixel (educator data) and prediction tap (student data) 
of the foreground used as an attention pixel. 

[0210] That is, the normal equation configuration section 62 performs the 
multiplication (xijxi'j') of the student data (prediction tap) used as the multiplier of a 
prediction coefficient in the left part of the normal equation of an equation (7), and the 
operation equivalent to a summation (sigma) using a prediction tap. 
[0211] Furthermore, the normal equation configuration section 62 performs the 
multiplication (xijyj) of the student data (prediction tap) in the right-hand side of the 
normal equation of an equation (7), and educator data (attention pixel), and the 
operation equivalent to a summation (sigma) using a prediction tap and an attention 
pixel. 

[0212] The normal equation having held as an attention pixel and having shown by this 
the pixel from which the above processing constitutes the foreground of the 1st frame 
thru/or each Nth frame at a ceremony (7) is built in the normal equation configuration 
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section 62. 

[0213] And the prediction coefficient calculation section 63 asks for a prediction 
coefficient, and makes the prediction coefficient storage section 46 ( drawing 24 ) 
supply and memorize after that the normal equation generated in the normal equation 
configuration section 62 by solution Lycium chinense. 

[0214] Here, with the gestalt of this operation, forecast [ of the pixel value An (x y) of 
the pixel which is in the location (x y) of n-th^frame standard coordinates' existence 
within the limits among the 1st frame in the condition of having performed alignment 
of a background thru/or the Nth frame ] E [An (x y)] is called for in the adaptation 
processing section 47 of drawing 24 for example, according to a degree type. 
[0215] 

E[An(x.y)]=g(F,B,E,n) 
... (8) 

Here, in a formula (8), F, B, and E express a front are recording image, a back are 
recording image, and an error image, respectively, and Function g (F, B, E, n) is defined 
by the degree type equivalent to the linearity linear expression of a formula (1 ). 
[0216] 

g(F, B, E, n) =wF1xf1+wF2xf2+ ... + wB1xb1+wB2xb2+ ... + wE1xe1+wE2xe2+ ... +wxn ... 
(9) 

Here, in a formula (9), wF1, wF2 wB1 and wB2 wE1, wE2 w express a 

prediction coefficient. Moreover, the pixel which constitutes the prediction tap about 
an attention pixel among the pixels from which e1, e2, and ... constitute the error 
image E for the pixel which constitutes the prediction tap about an attention pixel 
among the pixels from which b1, b2, and ... constitute the back are-recording image B 
for the pixel which constitutes the prediction tap about an attention pixel among the 
pixels from which f 1 , f2, and ... constitute the front are recording image F is expressed, 
respectively. 

[0217] When Function g (F, B, E, n) is defined by the equation (9), in the normal 
equation configuration section 62 of drawing 26 The normal equation for asking for the 

prediction coefficients wF1 and wF2 in an equation (9) wB1 and wB2 wE1, 

wE2 w is built. In the prediction coefficient calculation section 63 Prediction 

coefficients wF1 and wF2 wB1 and wB2 wE1 , wE2 w are asked for the normal 

equation by solution Lycium chinense. Therefore, the prediction coefficients wF1 and 

wF2 of one set wB1 and wB2 wE1 , wE2 w are called for about the foreground 

of the 1st frame thru/or all the Nth frame in this case. 

[0218] Next, drawing 27 shows the example of a configuration of the adaptation 
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processing section 47 of drawing 24 . 

[0219] The foreground are recording image memorized by the foreground are 
recording image storage section 44 and the error image further memorized by the 
error image storage section 49 if needed are supplied to the prediction tap 
configuration section 71. and the prediction tap configuration section 71 like the case 
in the prediction tap configuration section 61 of drawing 26 What is going to calculate 
a forecast among the pixels which constitute the foreground of the 1st frame thru/or 
each Nth frame in the condition of having performed alignment of a background is set 
to standard coordinates as an attention pixel. The pixel of the front are recording 
image in the location near an attention pixel and a space target and a back are 
recording image and the pixel of an error image are extracted, and it outputs to the 
prediction operation part 72 as a prediction tap. 

[0220] A prediction tap is supplied from the prediction tap configuration section 71, 
and also the prediction coefficient memorized by the prediction coefficient storage 
section 46 ( drawing 24 ) is supplied to the prediction operation part 72. And in the 
prediction operation part 72, the forecast of the pixel of a foreground used as an 
attention pixel is calculated by calculating the linearity linear expression defined by a 
formula (8) and (9) using a prediction tap and a prediction coefficient, and is outputted 
to the error count section 48. 

[0221] Next, with reference to the flow chart of drawing 28 , the foreground coding 
processing which encodes the foreground of the 1st frame thru/or each Nth frame 
performed in the foreground coding section 1 6 of drawing 24 is explained. 
[0222] It sets to step S161 first. The foreground extract section 41 The camera 
motion vector memorized by the camera motion vector storage section 1 3 ( drawing 
2 ), And as it mentioned above, a foreground is extracted, and the foreground storage 
section 42 is supplied and is made to memorize using the whole background 
memorized by the background memory 1 5 ( drawing 2 ) from each image of the 1 st 
frame thru/or the Nth frame memorized by the are recording section 1 1 ( drawing 2 ). 
[0223] And it progresses to step S162, and the foreground are recording image 
configuration section 43 constitutes a front are recording image and a back are 
recording image which were explained by drawing 25 from a foreground of the 1st 
frame memorized by the foreground storage section 42 thru/or each Nth frame, and 
the foreground are recording image storage section 44 is made to supply and 
memorize it, and it progresses to step 1 63. 

[0224] At step S1 63, study is performed using the pixel which constitutes the front 
are recording image memorized by the foreground are recording image storage section 
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44 in the study section 45 and a back are recording image, and the error image further 
memorized by the error image storage section 49 if needed, and, thereby, the 
prediction coefficient for predicting the pixel which constitutes the foreground of the 
1 st frame thru/or each Nth frame is called for. 

[0225] Here, when study processing is first performed at step S163, since the error 
image is not memorized by the error image storage section 49, study is still performed 
in it, without using an error image (it being unable to use). 

[0226] The prediction coefficient obtained as a result of the study in step S163 is 
supplied to the prediction coefficient storage section 46 from the study section 45, 
and is memorized in the form to overwrite. If a prediction coefficient is memorized by 
the prediction coefficient storage section 46, it will set to step S164. The adaptation 
processing section 47 To the pixel which constitutes the front are recording image 
and back are recording image which were memorized by the foreground are recording 
image storage section 44, the prediction coefficient memorized by the prediction 
coefficient storage section 46, and a pan Adaptation processing which calculates the 
forecast of each pixel which constitutes the foreground of the 1 st frame thru/or each 
Nth frame is performed by calculating a formula (8) and the linearity linear expression 
of (9) using the error image memorized by the error image storage section 49 if 
needed. 

[0227] Here, when adaptation processing is first performed at step S164, since the 
error image is not memorized by the error image storage section 49, adaptation 
processing is still performed in it, without using an error image (it being unable to use). 
[0228] The forecast of each pixel which constitutes the foreground of the 1st frame 
obtained as a result of the adaptation processing in step S164 thru/or each Nth frame 
is supplied to the error count section 48, in the error count section 48, in step S165, it 
is referring to the foreground of the 1st frame memorized by the foreground storage 
section 42 thru/or each Nth frame, and the prediction error of the forecast of each 
pixel which constitutes the foreground is searched for. 

[0229] And it progresses to step S166, and about each location of standard 
coordinates' existence within the limits, the absolute value of the prediction error in 
the location extracts the pixel which is max from the pixel which constitutes the 
foreground of the 1st frame thru/or each Nth frame, and suppjies the error count 
section 48 to the error image storage section 49 by using the image which becomes 
by the prediction error of the extracted pixel as an error image. In the error image 
storage section 49, the error image from the error count section 48 is memorized. 
[0230] Then, it progresses to step S167, total of the absolute value of the pixel value 
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(prediction error) of each pixel which constitutes the error image memorized by the 
error image storage section 49 in the error judging section 50 is called for, and it is 
judged whether the total is below a predetermined threshold (following). 
[0231] In step S167, when it judges that total of the absolute value of the pixel value 
(prediction error) of each pixel which constitutes an error image is not below a 
predetermined threshold (i.e., when the precision of the forecast of the foreground 
obtained from a prediction coefficient, and a front are recording image and a back are 
recording image (the need is accepted and it is an error image further) is not high), the 
same processing is repeated by step S163 return and the following. 
[0232] In addition, at next steps S163 or S164, since the error image is memorized by 
the error image storage section 49 in this case, as shown in drawing 29 , an error 
image besides a front are recording image and a back are recording image is also used, 
a prediction tap is constituted, and study processing or adaptation processing is 
performed, respectively. 

[0233] A prediction coefficient when it judges that total of the absolute value of the 
pixel value (prediction error) of each pixel which constitutes an error image is below a 
predetermined threshold in step S167 on the other hand, A front are recording image 
and a back are recording image (further) When the precision of the forecast of the 
foreground obtained from an error image if needed is high, it progresses to step S168. 
The error judging section 50 MUX51 is controlled, and the front are recording image 
memorized by the foreground are recording image storage section 44 and a back are 
recording image, the prediction coefficient memorized by the prediction coefficient 
storage section 46, and when the error image is memorized by the error image storage 
section 49, the error image is made to read and multiplex further. And MUX51 outputs 
the multiplexing data obtained as a result of the multiplexing to a multiplexer 17 
( drawing 2 ) as a coding result of the foreground of the 1st frame thru/or each Nth 
frame, and ends foreground coding processing. 

[0234] In addition, it may be made to make the new error image obtained whenever 
processing of step S166 is performed memorize in the form which overwrites the 
already memorized error image, and it leaves the already memorized error image to 
the error image storage section 49 as it is, and you may make it make it memorize it. 
[0235] When making a new error image memorize in the form which overwrites the 
error image already memorized by the error image storage section 49, although a 
certain amount of limitation is always in improvement in the precision of the forecast 
of a foreground since the number of error images is one even if study of a prediction 
coefficient is performed repeatedly, they can lessen the amount of data of 
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multiplexing data. 

[0236] Although the amount of data of multiplexing data increases somewhat on the 
other hand since an error image becomes two or more sheets in leaving the error 
image already memorized by the error image storage section 49 in a new error image 
as it is and making it memorize Since the error image of two or more sheets is used 
for a front are recording image and a back are recording image, and a list, a prediction 
tap is constituted and study processing or adaptation processing is performed, 
respectively as shown in drawing 30 , the precision of the forecast of a foreground can 
be raised more. In addition, drawing 30 shows the case where there are two error 
images. 

[0237] Moreover, although it asked for the prediction coefficient which calculates a 
forecast, also uses an error image when the prediction error is large, performs study 
processing again, and makes a prediction error small by performing adaptation 
processing using the prediction coefficient for which it asked by study processing in 
the above-mentioned case Irrespective of the size of a prediction error, it is also 
possible to make it output the prediction coefficient obtained as a result of study 
processing of the beginning by the study section 45 as a coding result of a foreground 
with the front are recording image and back are recording image which were 
memorized by the foreground are recording image storage section 44. In this case, the 
foreground coding section 1 6 is that the adaptation processing section 47, the error 
count section 48, the error image storage section 49, and the error judging section 50 
can be constituted without preparing. 

[0238] Furthermore, in an above-mentioned case, a formula (9) defines the function g 
(F, B, E, n) which calculates the forecast of a foreground, and although it asked for the 
prediction coefficient of one set used common to this calculating the forecast of the 
foreground of the 1 st frame thru/or all the Nth frame, in addition to this, a prediction 
coefficient can also be asked [ every frame and ] for every multiple frame. 
[0239] Namely, what is necessary is to define Function g (F, B, E, n), as shown in an 
equation (10), to build a normal equation for every frame, and just to make it ask for a 
prediction coefficient by solution Lycium chinense, in asking for a prediction 
coefficient for every frame. 
[0240] 

g(F, B. E, n) =wF1 nxf 1 +wF2nxf2+ ... + wB1 nxbl +wB2nxb2+ ... +wE1 nxel +wE2nxe2+ 

(10) 

Here, in a formula (10), the prediction coefficient used for wF1n, wF2n wB1n, 

wB2n wE1n, wE2n, and ... calculating the forecast of the foreground of the n-th 
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frame is expressed. 

[0241] Moreover, although the forecast of a foreground was calculated by linearity 
primary prediction, the forecast of a foreground can also be made to ask more than by 
the secondary high order prediction type in addition to this here. 
[0242] Furthermore, although it was made to learn the prediction coefficient for 
calculating the forecast of the foreground of the 1 st frame thru/or each Nth frame 
here using the front are recording image and back are recording image which 
consisted of foregrounds of the 1 st frame thru/or each Nth frame, it is also possible 
for it to be made to perform study of a prediction coefficient using images other than 
a front are recording image and a back are recording image. That is, study of a 
prediction coefficient can be made to be carried out by operating the pixel value of the 
image which becomes with the noise so that the prediction error of the forecast of a 
foreground may be made small using the image of one or more sheets which becomes 
with a noise. 

[0243] Next, drawing 31 shows the example of a configuration of the decoder 2 of 
drawing 1 . 

[0244] The coded data transmitted through a transmission medium 3 ( drawing 1 ) or 
the coded data reproduced from the record medium 4 ( drawing 1 ) is supplied to 
DMUX(demultiplexer) 81, and DMUX81 divides into a front are recording image, a back 
are recording image, a prediction coefficient, a whole background, and a camera 
motion vector the coded data supplied there. In addition, when an error image is 
contained in coded data, DMUX81 also separates the error image from coded data. 
[0245] Further, a front are recording image and a back are recording image, and when 
an error image is contained in coded data, the error image is supplied to the image 
storage section 86 from DMUX81. Moreover, a prediction coefficient, a whole 
background, or a camera motion vector is supplied to the prediction coefficient 
storage section 82, the background memory 87, and the camera motion vector storage 
section 88 from DMUX81, respectively. 

[0246] The prediction coefficient storage section 82 memorizes the prediction 
coefficient from DMUX81. The adaptation processing section 83 calculates the 
forecast of the foreground of the 1st frame thru/or each Nth frame by the front are 
recording image memorized by the prediction coefficient memorized by the prediction 
coefficient storage section 82 and the list at the image storage section 86 and the 
back are recording image, and performing the same adaptation processing as a case 
[ in / using an error image / corresponding to the need further / the adaptation 
processing section 47 of drawing 24 ]. 
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[0247] The foreground storage section 84 memorizes the forecast of the foreground 
of the 1st frame thru/or each Nth frame called for by the adaptation processing 
section 83 as a decode result of the foreground of the 1 st frame thru/or each Nth 
frame. 

[0248] The synthetic section 85 is based on the camera motion vector vn of the n-th 
frame memorized by the camera motion vector storage section 88 from the whole 
background memorized by the background memory 87. The background of the n-th 
frame is started (extracting), and by compounding the decode result of the 
background of the n-th frame, and the foreground of the n-th frame memorized by the 
foreground storage section 84, the image of the n-th frame is decoded and it outputs. 
[0249] The image storage section 86 memorizes an error image in the front are 
recording image supplied from DMUX81 and a back are recording image, and a list. The 
background memory 87 memorizes the whole background supplied from DMUX81. The 
camera motion vector storage section 88 memorizes the camera motion vector of the 
1st frame supplied from DMUX81 thru/or each Nth frame. 

[0250] Next, with reference to the flow chart of drawing 32 , the decode processing 
which decodes the image of the 1 st frame thru/or the Nth frame as a series of images 
performed in the decoder 2 of drawing 31 is explained. 

[0251] First, in step S1 71 , DMUX81 divides into a front are recording image, a back are 
recording image, a required error image, a prediction coefficient, a whole background, 
and a camera motion vector the coded data supplied there. A front are recording 
image, a back are recording image, and a required error image are supplied to the 
image storage section 86, and are memorized. Moreover, a prediction coefficient, a 
whole background, or a camera motion vector is supplied to the prediction coefficient 
storage section 82, the background memory 87, and the camera motion vector storage 
section 88, respectively, and is memorized. 

[0252] Then, it progresses to step S172, and using the prediction coefficient 
memorized by the prediction coefficient storage section 82, the front are recording 
image memorized by the list at the image storage section 86, a back are recording 
image, and a required error image, the adaptation processing section 83 is performing 
the same adaptation processing as the case in the adaptation processing section 47 
of drawing 24 , and calculates the forecast of the foreground of the 1 st frame thru/or 
each Nth frame. This forecast is supplied to the foreground storage section 84, and is 
memorized as a decode result of the foreground of the 1 st frame thru/or each Nth 
frame. 

[0253] And it progresses to step S173, and in the synthetic section 85, based on the 
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camera motion vector vn of the n~th frame memorized by the camera motion vector 
storage section 88, the background of the n-th frame is started from the whole 
background memorized by the background memory 87, and the decode result of the 
background of the n-th frame and the foreground of the n-th frame memorized by the 
foreground storage section 84 is compounded. In the synthetic section 85, the above 
processing is performed about the 1st frame thru/or all the Nth frame, and ends 
decode processing. 

[0254] Next, hardware can also perform a series of processings mentioned above, and 
software can also perform. When software performs a series of processings, the 
program which constitutes the software is installed in the computer built into the 
encoder 1 and decoder 2 as hardware of dedication, or the general-purpose computer 
which performs various kinds of processings by installing various kinds of programs. 
[0255] Then, the medium used in order to install in a computer the program which 
performs a series of processings mentioned above and to make it into the condition 
which can be performed by computer with reference to drawing 33 is explained. 
[0256] As shown in drawing 33 (A), a user can be provided with a program in the 
condition of having installed in the hard disk 102 and semiconductor memory 103 as a 
record medium which are built in the computer 101 beforehand. 

[0257] Or as shown in drawing 33 (B), a program can be stored in record media, such 
as a floppy disk 111, CD-ROM (Compact Disc Read Only Memory)112, the MO 
(Magneto optical) disk 113, DVD (Digital Versatile Disc)1 14, a magnetic disk 115, and 
semiconductor memory 116, temporarily or permanently, and can be offered as a 
software package again. 

[0258] Furthermore, it transmits to a computer 101 on radio, or a program is 
transmitted to a computer 123 with a cable through the networks 131, such as LAN 
(Local Area Network) and the Internet, and can be made to store in the hard disk 1 02 
to build in in a computer 101 through the satellite 122 for digital satellite broadcasting 
services from the download site 121, as shown in drawing 33 (C). 

[0259] The medium in this description means the concept of the wide sense 
containing all these media. 

[0260] Moreover, it is not necessary to necessarily process the step which describes 
the program offered by the medium to time series in accordance with the sequence 
indicated as a flow chart, and it is a juxtaposition thing also including the processing 
(for example, parallel processing or processing by the object) performed according to 
an individual in this description. 

[0261] Next, drawing 34 shows the example of a configuration of the computer 101 of 
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drawing 33 . 

[0262] The computer 101 contains CPU (Central Processing Unit)142, as shown in 
drawing 34 . The input/output interface 145 is connected to CPU 142 through the bus 
141, and CPU142 will perform the program stored in ROM (Read Only Memory)143 
corresponding to the semiconductor memory 103 of drawing 33 (A) according to it, if a 
command is inputted when the input section 147 which consists of a keyboard, a 
mouse, etc. is operated by the user through an input/output interface 145. Or it is 
transmitted from the program and satellite 122 with which CPU142 is stored in the 
hard disk 102 again, or a network 131, and the program which reading appearance was 
carried out from the program which was received in the communications department 
148 and installed on the hard disk 102 or the floppy disk 1 1 1 with which the drive 149 
was equipped, CD-ROM112, MO disk 113, DVD114, or the magnetic disk 115, and was 
installed on the hard disk 102 is loaded to RAM (Random Access Memory)144, and is 
performed. And CPU142 outputs the processing result to the display 146 which 
consists of LCD (Liquid CryStal Display) etc. through an input/output interface 145 if 
needed. 
[0263] 

[Effect of the Invention] The inside of the are recording image memorized by are 
recording image storage means to memorize the are recording image in the condition 
of having piled up by performing alignment of the background of each screen of the 
image which image pick-up equipment outputs in the motion detection equipment of 
this invention and the motion detection approach, and a list according to the medium. 
The detection range used for detecting the motion about the attention screen 
containing the image of the screen in front of an attention screen is set up, and the 
center of gravity of the detection range and the center of gravity of an attention 
screen are computed. And based on the center of gravity of the detection range, and 
the center of gravity of an attention screen, the motion vector showing a motion of 
the image pick-up direction of the image pick-up equipment when picturizing the 
image of an attention screen is called for, and the image of an attention screen is 
written in an are recording image storage means based on the motion vector. 
Therefore, it becomes possible to ask for a motion of the image pick-up direction of 
the image pick-up equipment at the time of the image pick-up of an image simply. 
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[Brief Description of the Drawings] 

[Drawing l] It is drawing showing the example of a configuration of the gestalt of 1 
operation of the picture transmission equipment which applied this invention. 
[Drawing 2] It is the block diagram showing the example of a configuration of the 
encoder 1 of drawing 1 . 

[Drawing 3] It is drawing showing a series of images memorized by the are recording 
section 11 of drawing 2 . 

[Drawing 4] It is drawing for explaining processing of the camera motion detecting 
element 12 of drawing 2 . 

[Drawing 5] It is the block diagram showing the example of a configuration of the 
camera motion detecting element 12 of drawing 2 . 

[Drawing 6] It is drawing for explaining how asking for the camera motion vector by the 
camera motion detecting element 12 of drawing 5 . 

[Drawing 7] It is a flow chart for explaining processing (camera motion detection 
processing) of the camera motion detecting element 12 of drawing 5 . 
[Drawing 8] It is a flow chart for explaining the 1st example of the detail of the 
processing (center-of-gravity calculation processing of an attention frame) in step S3 of 
drawing 7 . 

[Drawing 9] It is drawing showing a level table. 

[Drawing 10] It is a flow chart for explaining the detail of the processing 
(center-of-gravity calculation processing of the motion detection range) in step S4 of 
drawing 7 . 

[Drawing ll] It is a flow chart for explaining the 2nd example of the detail of the 

processing in step S3 of drawing 7 . 

[Drawing 12] It is drawing showing a class table. 

[Drawing 13] It is a flow chart for explaining the 3rd example of the detail of the 
processing in step S3 of drawing 7 . 

[Drawing 14] It is drawing for explaining an edge pixel. 

[Drawing 15] It is a flow chart for explaining the detail of processing (motion-less 
judgment processing) of the motion-less judgment section 25 of drawing 5 . 
[Drawing 16] It is the block diagram showing the example of a configuration of the 
background extract section 14 of drawing 2 . 

[Drawing 17] It is drawing for explaining processing of the drawing 16's existence range 
detecting element 31. 

[Drawing 18] It is drawing for explaining processing of the read-out section 32 of 
drawing 16 . 
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[Drawing 19] It is drawing showing a frequency table. 

[Drawing 20] It is a flow chart for explaining processing (background extract 
processing) of the background extract section 14 of drawing 16 . 

[Drawing 21] It is a flow chart for explaining the detail of processing (background pixel 
extract processing) of step S124 of drawing 20 . 

[Drawing 22] It is a flow chart for explaining the detail of processing (background 
escape processing) of step S125 of drawing 20 . 

[Drawing 23] It is drawing for explaining background escape processing of drawing 22 . 
[Drawing 24] It is the block diagram showing the example of a configuration of the 
foreground coding section 16 of drawing 2 . 

[Drawing 25] It is drawing for explaining processing of the foreground are recording 
image configuration section 43 of drawing 24 . 

[Drawing 26] It is the block diagram showing the example of a configuration of the 
study section 45 of drawing 24 . 

[Drawing 27] It is the block diagram showing the example of a configuration of the 
adaptation processing section 47 of drawing 24 . 

[Drawing 28] It is a flow chart for explaining the detail of processing (foreground coding 
processing) of the foreground coding section 16 of drawing 24 . 

[Drawing 29] A prediction tap is drawing showing signs that it is constituted using an 
error image. 

[Drawing 30] A prediction tap is drawing showing signs that it is constituted using the 
error image of two sheets. 

[Drawing 31] It is the block diagram showing the example of a configuration of the 
decoder 2 of drawing 1 . 

[Drawing 32] It is a flow chart for explaining processing (decode processing) of the 
decoder 2 of drawing 31 . 

[Drawing 33] It is drawing for explaining the medium which applied this invention. 
[Drawing 34] It is the block diagram showing the example of a configuration of the 
computer 101 of drawing 33 . 
[Description of Notations] 

1 Encoder 2 Decoder, 3 Transmission medium 4 A record medium, 11 are-recording 
section 12 A camera motion detecting element, 13 The camera motion vector storage 
section and 14 background extract section 15 Background memory, 16 Foreground 
coding section 17 MUX, 21 Center-of-gravity calculation section 22 A vector detecting 
element, 23 A write-in control section, 24 An are recording image memory, 25 
Motion-less judgment section 31 An existence range detecting element, 32 Read-out 
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section 33 Frequency count area 34 The frequency table storage section and 35 The 
write-in section 36 Background flag memory 41 The foreground extract section, 42 
foreground storage section 43 The foreground are recording image configuration section, 
44 foreground are recording image storage section, 45 The study section, 46 Prediction 
coefficient storage section 47 The adaptation processing section, 48 Error count section 
49 The error image storage section, 50 Error judging section 51 MUX, 61 prediction tap 
configuration section 62 The normal equation configuration section, 63 The prediction 
coefficient calculation section, 71 The prediction tap configuration section, 72 Prediction 
operation part 81 DMUX, 82 Prediction coefficient storage section 83 The adaptation 
processing section, 84 Foreground storage section 85 The synthetic section, 86 image 
storage section 87 Background memory 88 The camera motion vector storage section, 
101 Computer 102 Hard disk 103 Semiconductor memory, 111 Floppy disk 112 CD-ROM, 
113 An MO disk, 114 DVD, 115 Magnetic disk 116 Semiconductor memory, 121 
download site 122 satellites, 131 Network 141 buses 142 CPU 143 ROM 144 RAM 145 
Input/output interface 146 Display 147 Input section 148 Communications department 
149 Drive 
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